User interface for summarizing the relevance of a document to a query

ABSTRACT

According to an aspect, summarizing relevance of a document to a conceptual query includes receiving the conceptual query, accessing concepts extracted from the document, and computing a degree to which the conceptual query is related to each of the extracted concepts. The computing is responsive to a metric that measures a relevance between the concepts in the conceptual query and the extracted concepts. The method also includes creating a summary by selecting a threshold number of the concepts having a greatest degree of relation to the conceptual query, and outputting the summary including the selected threshold number of concepts.

DOMESTIC PRIORITY

This application is a continuation of U.S. patent application Ser. No.14/440,461, filed Jul. 14, 2014, the content of which is incorporated byreference herein in its entirety.

BACKGROUND

The present disclosure relates generally to semantic searching, and morespecifically, to summarizing the relevance of documents to a queries.

Traditional information retrieval technologies are based on the indexingof data using keywords. A drawback of using keywords to index data isthat when a query keyword is not present in a document, the document isnot considered for a possible match to the query. One approach that hasbeen employed to improve this situation is the application of queryexpansion techniques in which the original query is substituted withvariants. When using query expansion techniques, a search term issubstituted with a synonym, independent searches are performed on thesearch term and the synonym, and then the searches subsequently joined.In principle, the larger the number of different variants that are triedat query time, the higher the potential quality of the returned results,assuming that a good joining technique is in place. One disadvantage toquery expansion techniques is that the need for very fast response timesimplies that these techniques need to be relatively inexpensive, oralternately, that there are significant computational resources devotedto a search query so as to be able to complete the independent querieson time. This disadvantage is made particularly acute when doingsearches with extremely rich ontologies because the query expansion maytake a single search and turn it into potentially tens of thousands ofsearches.

Within the field of semantic reasoning techniques, one of the mostpopular techniques is latent semantic analysis (LSA). LSA functions byprojecting a document's representation, typically in the form of aterm-frequency/inverse-document-frequency vector, to a smaller spacecalled the latent semantic space. This projecting is performed usingmatrix factorization techniques such as singular value decomposition(SVD), or by using statistical inference techniques such asexpectation/maximization, as used in probabilistic latent semanticindexing (PLSI). Every dimension in this lower dimensional space ismeant to represent an abstract concept that is generated automaticallyby the LSA technique. The LSA idea has given rise to numerous othervariants that generally share the attributes described above. Adisadvantage of the LSA family of techniques is that by themselves, theyare not able to take advantage of the large volumes of crowd sourceddata that have become available through the popularity of websites suchas Wikipedia.

SUMMARY

Embodiments include a method for summarizing the relevance of a documentto a conceptual query. A method includes receiving the conceptual query,accessing concepts extracted from the document, and computing a degreeto which the conceptual query is related to each of the extractedconcepts. The computing is responsive to a metric that measures arelevance between the concepts in the conceptual query and the extractedconcepts. The method also includes creating a summary by selecting athreshold number of the concepts having a greatest degree of relation tothe conceptual query, and outputting the summary including the selectedthreshold number of concepts.

Another embodiment includes a method for receiving the conceptual query,accessing extracted concepts for each of the documents, and computing adegree to which each of the documents are related to one another. Thecomputing is responsive to a metric that measures relevance between theextracted concepts in one document and extracted concepts in anotherdocument. The method further includes assigning the documents to one ormore groups based on the computing. A pair of documents having a firstscore that specifies a degree of relation is more likely to be in thesame group than a pair of documents having a second score specifying adegree of relation that is lower than the first score. The method alsoincludes outputting results of the assigning.

Additional features and advantages are realized through the techniquesof the present disclosure. Other embodiments and aspects of thedisclosure are described in detail herein. For a better understanding ofthe disclosure with the advantages and the features, refer to thedescription and to the drawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The subject matter which is regarded as the invention is particularlypointed out and distinctly claimed in the claims at the conclusion ofthe specification. The forgoing and other features, and advantages ofthe invention are apparent from the following detailed description takenin conjunction with the accompanying drawings in which:

FIG. 1 depicts a high level view of a system for performing semanticsearching in accordance with an embodiment;

FIG. 2 depicts a process for automatically linking text to concepts in aknowledge based in accordance with an embodiment;

FIG. 3 depicts a user interface for displaying search results inaccordance with an embodiment;

FIG. 4 depicts a process for new concept definition in accordance withan embodiment;

FIG. 5 depicts a process for computing the relevance of a document toconcepts not specified in the document in accordance with an embodiment;

FIG. 6 depicts a process for storing and querying conceptual indices inan inverted table in accordance with an embodiment;

FIG. 7 depicts an external view of a user interface of a researcherprofile in accordance with an embodiment;

FIG. 8 depicts a user interface of an external view of a researcherpublication page in accordance with an embodiment;

FIG. 9 depicts a user interface of an internal view of a portion of aresearcher patent page in accordance with an embodiment;

FIG. 10 depicts a user interface of a portion of a researcher projectpage in accordance with an embodiment;

FIG. 11 depicts a user interface of a primary profile editor inaccordance with an embodiment;

FIG. 12 depicts a user interface for a portion of an editor for aresearcher publication page in accordance with an embodiment;

FIGS. 13A, 13B, and 13C depict a user interface for searching andreceiving search results in accordance with an embodiment;

FIG. 14 depicts a user interface for displaying a result set inaccordance with an embodiment;

FIG. 15 depicts a user interface for displaying search results inaccordance with an embodiment;

FIG. 16 depicts a user interface for displaying a portion of searchresults in accordance with an embodiment;

FIG. 17 depicts a user interface in accordance with an embodiment;

FIG. 18 depicts a user interface for displaying a portion of searchresults in accordance with an embodiment;

FIG. 19 depicts a user interface for displaying a portion of searchresults in accordance with an embodiment;

FIG. 20 depicts a high-level block diagram of a question-answer (QA)framework where embodiments of semantic searching can be implemented inaccordance with an embodiment; and

FIG. 21 depicts a processing system for performing semantic searching inaccordance with an embodiment.

DETAILED DESCRIPTION

Embodiments described herein can be utilized for searching,recommending, and exploring documents through conceptual associations.Embodiments include the use of semantic search technologies that combineinformation contained in knowledge bases and unstructured data sourcesto provide improved search results.

Embodiments described herein can also be utilized to automatically linktext (e.g., from a document) to concepts in a knowledge base. Forexample, the sentence “The computer programmer learned how to write Javain school.” contains three concepts: computer programmer, Java, andschool. An embodiment of the automatic text linker described herein candiscover these three concepts and link them to the most relevantconcepts that it can find in a knowledge base.

Embodiments described herein can further be utilized to automaticallyadd new entries into an existing knowledge base based on contents of adocument. This can allow the knowledge base to continue to be updatedand remain relevant as new documents are added to a corpus.

Embodiments described herein can further be utilized to compute therelevance of a document to concepts that are not specified in thedocument. Concepts extracted from the document can be used along with aconcept graph to determine whether the document is related to otherconcepts.

Embodiments described herein can include the use of efficient datastructures for storing and querying deep conceptual indices. A documentcan be received, concepts extracted from the document, confidence levelscalculated for the various concepts, and then a representation of thedocument can be created in a concept space that connects the document toall possible concepts (not only the ones that were found explicitly inthe document). Embodiments described herein are directed to how thisinformation can be organized in a computer system so that it can beefficiently queried against and maintained.

Embodiments described herein can further include providing a userinterface for summarizing the relevance of a document to a conceptualquery. In addition the relevance of a plurality of documents to aconceptual query can be summarized, including assigning the documents togroups based on how relevant the documents are to concepts specified inthe query.

As used herein, the term “concept” refers to an abstract idea or generalnotion that can be specified using a collection of names or labels, anda corresponding description. Additionally, sample sentences describingthe concept may be included in the description of the concept. Concepts,such as, for example, “To be or not to be”, “singular valuedecomposition”, “New York Yankees” or “iPhone 6” may be encoded in a webpage (e.g., Wikipedia).

As used herein the term “query” refers to a request for information froma data source. A query can typically be formed by specifying a conceptor a set of concepts in a user interface directly or indirectly bystating a query in natural language from which concepts are thenextracted. As used herein the term “conceptual query” refers to a typeof query that is specified by listing one or more concepts in a conceptgraph.

As used herein the term “corpus” refers to a collection of one or moredocuments represented using text (e.g., unstructured text).

As used herein, the term “concept graph” refers to a visualrepresentation that may be used to define a space of concepts and theirrelationships. A concept graph is an example of a knowledge base inwhich knowledge is represented with nodes in the graph as concepts andedges in the graph representing known relations between the concepts. Aconcept graph can be derived from crowd-sourced data sources such aswikis, which focus on defining concepts (e.g., Wikipedia). A conceptgraph can additionally be augmented with concepts found in newunstructured data sources.

Embodiments described herein rely on the ability to compute an estimateof how relevant a concept in a concept graph is to another concept inthe concept graph. One way to accomplish this is through the use ofMarkov chain techniques. A Markov chain can be built by regarding eachof the nodes in the concept graph as a state in the Markov chain, wherethe links in the concept graph are an indication of the next states thatmay be visited from that state. The probability of going to a stateconditional upon being in another state can be made to depend on aweight that may exist for an edge in the concept graph. Suppose thereare two concepts, A and B, and a requirement to compute how relevantthese concepts are to each other. An initialization probability vectoris initialized by setting to 1.0 the probability of being in the staterelated to concept A, and then the Markov chain is iterated.Computationally, this is accomplished by first multiplying thetransition probability matrix of the Markov chain times the currentstate probability vector. After iterating, the resulting vector islinearly mixed with the initialization probability vector so as toemulate a “teleportation” back to the initial concept. Mathematically,if the initialization probability vector is v0, and the transitionprobability matrix is M, and the teleportation parameter is alpha,iterating the Markov chain can mean computing this recursion:v^{i+1}=alpha M v^{i}+(1-alpha)v^0

This recursion is iterated for, for example, L steps. The resultingvector can be regarded as a measure of how relevant concept A is to allother concepts in the graph. However, experimentation with thistechnique shows that it often overstates or understates the relevance ofa concept to concept A in the following way: if a concept has many “inlinks” in the concept graph, (for example the concept is: U.S.A) then itwill be deemed highly relevant to concept A independently of whatconcept A is. A similar statement can be made when a concept A is verylightly linked. Therefore, v^L on its own may not suffice, and insteadit is processed through a normalization stage. One normalization is todivide each entry of v^L by the corresponding entries of a vector u^L,which is obtained by a similar recursion:u^{i+1}=alpha M u^{i}+(1-alpha)u^0where instead u^0 is a uniform distribution. This is an example of onetechnique and other normalization techniques are also possible.

Effectively, (setting aside the normalization described above) whatthese Markov chain techniques are accomplishing is to measure therelevance of a concept to another concept by performing a weighting ofeach of the paths that connect a concept to another concept. The scoreof each path can depend on the length (number of hops) of the path, onthe various transition probabilities connecting the concepts present inthe paths, and on the teleportation parameter alpha. The Markov chaincomputation has the advantage of performing this weighting in anefficient form relating concept A to every other concept. This techniquefor computing the degree to which a concept is relevant to anotherconcept described above is not the only possible technique. For example,a concept graph may have links which have type (or equivalentlypredicate) information, effectively creating (subject, predicate,object) triples. The link type may then be used to compute an additionallink weight, which can be used to modify the link's probability in thetransition probability matrix (by, for example, making the probabilityof the link proportional to the link weight and) which then affects theoverall path score. One possibility is to make the link weight for somelink types equal to zero, thereby effectively erasing the link from thegraph. The capability of affecting the path scores using link types andweights allows the definition of a family of techniques for measuringthe relevance of a concept to another concept, instead of a single one.Embodiments described herein may utilize each or a combination of thesetechniques.

Additional techniques that can be utilized by embodiments for computingthe relevance of one concept to another concept can include the use ofnot only the links between edges, but also information in the nodes ofthe concept graph such as the name or names of the concept and thedescription of the concept in order to measure the relevance to anotherconcept.

Additional mechanisms that can be utilized by embodiments for computingthe relevance of a concept to another concept include other knownsemantic similarity or relatedness techniques. Examples of suchtechniques can be found, for example, in articles such as: “UsingInformation Content to Evaluate Semantic Similarity in a Taxonomy” byPhilip Resnik (1995); “An Information-Theoretic Definition ofSimilarity” by Dekang Lin (1998); “Algorithmic Detection of SemanticSimilarity”, by Ana Gabriela et al. (2005); “Semantic Similarity Basedon Corpus Statistics and Lexical Taxonomy” by J. J. Jiang et al. (1997);“Introduction to Latent Semantic Analysis” by Landauer, Foltz, Laham(1998); “A Knowledge-Based Clustering Algorithm Driven by Gene Ontology”by Cheng, J et al. (2004); and “The Google Similarity Distance” byCilibrasi et al. (2007). It will be understood that embodiments remainapplicable and may utilize any metric for measuring the relevance of aconcept to another concept. Embodiments are not limited to specificconcept to concept degree of relevance or relatedness metrics.

Referring now to FIG. 1, a high level view of a system for performingsemantic searching is generally shown in accordance with an embodiment.The system shown in FIG. 1 can be utilized for searching, recommending,and exploring documents through conceptual associations. As shown in theembodiment of FIG. 1, a document 102 (e.g., from a corpus of documents)is input to a natural language processing (NLP) engine 104 and text fromthe document 102 is automatically linked to concepts in a knowledge base106. In an embodiment, the NLP engine 104 can also define new conceptsto be added to the knowledge base 106 based on contents of the document102. The concepts from the knowledge base (both existing and newlydefined if any) that are found in the document 102 are referred toherein as the “extracted concepts” or as the “concepts extracted fromthe document.”

As shown in block 108, the system can also compute the relevance of thedocument 102 to concepts in the knowledge base 106 that are notspecified in the document 102 using the extracted concepts and theknowledge base 106. In an embodiment, a degree of association betweenthe extracted concepts other concepts (e.g., as indicated by theknowledge base 106) is used to determine a relevance of the document 102to concepts not extracted from the document 102. Output from block 108can include, for each concept in the knowledge base 106, a likelihoodthat the document is related to the concept. In another embodiment, theoutput from block 108 can include the relation information for justthose concepts in the knowledge base 106 that meet a threshold such asthe “most related” or those with a likelihood over a specifiedthreshold.

The system can also generate a reverse index 110 that includes for eachconcept in the knowledge based 106, a likelihood that the document 102is related to the concept. The document scores shown in the embodimentof the reverse index 110 in FIG. 1 indicate the likelihood that thedocument is related to the concept. The system can also generate anexplanations index 114 which can be used to summarize the relevance ofthe document 102 to a query. After documents have been ingested andprocessed, and the reverse index 110 and explanations index have beencalculated, a query is the main mechanism with which an agent externalto our system interacts with our system. A query is simply some inputthat is passed to the system so that relevant documents are returned andsuggested, together with an explanation of why they are so. Most of thequeries described herein are “conceptual queries”, in which the queryincludes one or more concepts in knowledge base or concept graph.However conceptual queries may have been derived from a simple string oftext as well.

Embodiments described herein can be used to automatically link text(e.g., from a document) to concepts in a knowledge base. As describedabove, the sentence “The computer programmer learned how to write Javain school” contains three concepts: computer programmer, Java, andschool. An embodiment of the automatic text linker described herein candiscover these three concepts and link them to the closest (mostrelevant) concepts that it can find in a knowledge base. As used herein,the term “closest concept” refers to a concept that is “most relevant.”

In an embodiment it is assumed that natural language can be modeledusing a conceptual generative language model. A generative model is onewhich describes how to randomly generate data given some hiddenparameters. In a conceptual generative language model, the key mechanismfor generating output relies on the notion of a concept. In a conceptualgenerative model, it is assumed that that a human externalizes asequence of words that come to mind while thinking of a concept or acollection of concepts. For example, while thinking of the concepts “NewYork”, “Statue of Liberty” and “Yankees Stadium” one may produce thewords “Both icons of New York City, the State of Liberty and the YankeesStadium have attracted masses of visitors over the years.” As describedfurther herein, a conceptual generative model can be specified byconsidering concepts as hidden parameters and the output text as theobservations of a hidden Markov model.

In addition, in an embodiment, it can be assumed that the task ofdeciding whether to link a portion of text to a concept in a knowledgebase of concept graphs can be treated as a “differential” test; that is,some text is linked to a concept only if the probability of the textassuming that the author was “thinking” about that concept when writingthe text is sufficiently higher than the probability that the author wasthinking of no particular concept (i.e., a generic language model) whenwriting the text. In addition, a higher level of confidence can beassociated with linking a portion of text to a particular concept whenthe probability of the text assuming that the author was “thinking”about that concept is sufficiently higher than the probability that theauthor was thinking of other competing concepts. For example, thesentence “I wish I could drive a Maserati” will have a much higherprobability p1 of having been produced by someone thinking of theconcept “Maserati” than the probability p2 of being produced by someonewho is not thinking about any particular concept, the latter modeledtypically by assuming that words or collections of words are produced bypicking them through some random mechanism from a generic corpus oftext. Additionally, the same sentence has also a probability p3 of beingproduced by someone thinking of the concept “Fiat”, and that similarly,p3 will also be much higher than p2. So both “Maserati” and “Fiat” arecompeting concepts for what a person may have been thinking of whenhaving uttered that sentence. However, the probability p1 for “Maserati”would be noticeably higher than that of p3 (for “Fiat”) because it is amuch more specific concept for what the human may have been thinking ofat the time of utterance. So in the basis of this differential analysis,it can be concluded that one should select “Maserati” as the underlyingconcept, and possibly in addition, to point out that there is a word(the last word in the sentence) that can be pointed to as a specificmention of the concept “Maserati”. If the original sentence was changedto “I wish I could drive a car”, then while the probabilities p1 (forthinking of “Maserati”) and p3 (for thinking of “Fiat”) as defined abovewould still be high, since, the probability p2 of not thinking of anyparticular concept would be closer to p1 and p3, and p1 and p3 would becloser to each other. Therefore, on the basis of this new differentialanalysis, for this new sentence it can be determined that neitherMaserati nor Fiat would get a confidence of being what a human may havebeen thinking of when uttering that second sentence.

Sometimes, the differential analysis described above is not sufficientto establish a contrast between the various probabilities to make adecision, and additional information is needed. For example, in thesentence “I love me a good ball game” (never mind the incorrect grammarof the sentence), possible candidate competing concepts may be“baseball”, “soccer”, “tennis”, “basketball”, etc. Imagine however thatone was giving additional context, either in the form of a second textor a collection of concepts. For example, suppose it is known that theunderlying user had purchased items from a store specializing in NewYork Yankees memorabilia. Then, through analysis of the visited website,one can extract the concept “New York Yankees” as a possible sidecontext. When combining this information with the above, then thedifferential analysis improves, and then one might select “baseball” asa more likely concept.

In an embodiment, it can be assumed that when a human is thinking of aconcept, in his/her mind there are multiple data sources that relate tothis concept. For example, a set of data sources can include: one datasource that is a collection of names for the concept; another datasource that is a text that describes what this concept is; and a furtherdata source that is a series of examples to which the concept isreferred. Other sets of data sources may be used by embodiments and mayvary based on particular applications and concepts being analyzed.

In a generative model, it can be assumed that, with some probability,that a human will utter words that will either appear verbatim or withsome variation in one or more of the data sources in a set of datasources. In addition, it can be assumed that, with some probability, thehuman will utter a word that comes from a data source that is generic(e.g., not in the set of data sources for the concept), that is, it doesnot refer to any concept specifically. In the simplest generativemodels, each of the data sources in the example set of data sourcesdescribed above is associated with a bag-of-words model where a word ischosen independently and identically distributed from the bag-of-words.In addition, it can be assumed that each of the data sources is chosenwith a certain probability in an independent, identically distributedfashion.

More complex models for generating words can also be used to provideimproved linking between a portion (e.g., one or more words) of text anda concept. For example, when the data source is a set of names of aconcept, the model for selecting words can include the following: firstselect a name to be uttered with some probability; and then utter thesequence of words as given by the concept name selected. Additionalvariants can include one or more words in the sequence of words in theconcept name being skipped, or morphed into a variant (for example, adda plural or convert into singular, or capitalize, or lower case, or adda typo). A further variant includes the possibility of uttering thewords in the sequence of words out of order.

When uttering words from a data source that is the description of aconcept, a word in the data source can be selected at random using aprobability distribution that gives a higher probability to words thatoccur earlier in the description and a lower probability to words thatoccur later in the description. One possible model is to choose apositive integer x from a distribution that assigns a probability to xthat is proportional to (1/x)^(a) up to a maximum possible integer whichis the count of words in a document, where a is a parameter that can beestimated through standard statistical parameter estimation methods.After choosing this integer, the corresponding word is uttered. Afterthis, the simplest model simply repeats this process again. More complexmodels may choose a “phrase length” which results in uttering not onlythe word in position x, but also subsequent words in the order in whichthey appear in the data source up to a certain phrase length. Othermodels may not utter exactly the same words, but variants or synonymsthat can be obtained through the use of thesauri or other data sources.

The example in the above text describes how individual data sourcemodels can generate language, and suggests that the selection of whichdata source models are responsible for uttering language should be leftto a simple independent, identically distributed model. In anembodiment, this simple independent, identically distributed model isreplaced by a more complex Markov model where once a word is utteredfrom a data source, the chances that it is uttered from the same datasource are higher than the chances that it is uttered by a differentlanguage model.

The possibility that a person is thinking of more than one concept whenuttering language can also be considered. In this case, a sentence, orcollection of sentences, being uttered is regarded as having “epochs”inside of it. The epochs can be generated using a single concept model,however switching from an epoch to another epoch can result in languagebeing uttered from a different single concept model. The model forchoosing which concept to utter words from can also be treated as aMarkov model, where the chances that a word is uttered from a certainconcept after uttering a word coming from that concept's data sources ishigher than the chances of uttering a word from another concept's datasources.

The model described above can be seen as a form of a hierarchical hiddenMarkov model, where in the top level, a concept is chosen, then in thenext level, a data source within the concept is chosen, and then in thenext level, a word from the data source is uttered. The observables ofthis hierarchical hidden Markov model are the words generated and theun-observables are the various choices described above leading to theselection of the word.

As described herein, it is possible to model language utterances throughconceptual generative language models. Each of these conceptualgenerative language models can be associated with a technique forestimating the probability that a given sequence of words (e.g., asentence) is uttered for the given conceptual generative language model.These probability estimates can be obtained using computationaltechniques that will reflect the underlying conceptual generativelanguage model; for example for simple bag-of-words, counting thefrequency of the words in a given sentence and then using theprobability of a word in the bag-of-words suffices to compute aprobability estimate. In the case of a hidden Markov model (HMM), thesituation is more complex because to compute the exact probability givena model it can be necessary to sum the probabilities of all possiblesequences of states. The computational technique used in this case fallsunder the general class of dynamic programming methods; and it isimportant to note that in most instances, an exact calculation of theprobability is unnecessary and this can be used to further simplify thecalculations.

The idea of “differential” testing of probability assignments toportions of text is now discussed as it is a key idea that can beutilized for performing automatic linking(s) of text to concepts. As anexample, consider the previous example text “The computer programmerlearned how to write Java in school.”

Suppose that one is attempting to decide whether the word “Java” in thesentence above should be linked at all, and if so, whether it should belinked to Java the computer programming language, Java the Indonesianisland, or Java the coffee.

Using the previous discussion, for this particular task one can employfour different language models: 1) a generic language model in which noparticular specific concept is on the mind of the human (for exampleobtained by analyzing large volumes of text with a variety ofprovenances); 2) a language model for when Java the computer programminglanguage is being thought of; 3) a language model for when Java theIndonesian island is being thought of; and 4) a language model for whenJava the coffee bean is being thought of.

The probabilities of the term Java in the text T=“The computerprogrammer learned how to write Java in school” referring to aprogramming language, coffee bean, Indonesian island, or something elsecan be estimated using the four language models above. These Probabilityestimates can be written as:

p(T|Java the programming language)

p(T|Java the coffee bean)

p(T|Java the island)

p(T|generic language model)

Next, the following ratios can be formed:

p(T|Java the programming language)/p(T|generic language model)

p(T|Java the coffee bean)/p(T|generic language model)

p(T|Java the Island)/p(T|generic language model)

Only if any of these ratios is sufficiently high would linking the wordJava in the text to one of the three Java concepts be considered.

It can also be assumed that there could be a prior P(Java theprogramming language), P(Java the coffee bean), P(Java the island)denoting the a priori probability with which these concepts are expectedto show up in a pool of concepts extracted from text. Note that althoughthis prior probability should not favor any specific concept, it couldfavor concepts in a specific pool of concepts in a certain area oftopics (for example emphasizing technology). If no prior probability isidentified, it can be assumed that these probabilities are identical toeach other.

Next, a maximum can be chosen amongst:

p(T|Java the programming language)

p(Java the programming language)

p(T|Java the coffee bean) p(Java the coffee bean)

p(T|Java the island) p(Java the island)

This maximum can be regarded as a maximum a posteriori (MAP) estimate ofthe originating concept, or a maximum likelihood (ML) estimate if theprior probability is uninformative. From the earlier description, itstands to reason that p(T I Java the programming language), for example,would be higher than p(T|Java the coffee bean) since in the languagemodel for Java the computer programming language there will be a datasource containing a description where words like “computer”, “computerprogramming”, “computer programmer”, “programming”, and so on appear,while in the case of the language model for Java the coffee bean, thosewords will not appear.

If the maximum (in this case p(T|Java the programming language)) issufficiently higher than p(T|Java the coffee bean) and p(T|Java theIsland) and also if p(T|Java the programming language) is sufficientlyhigher than p(T|generic language model), where sufficiently higher canbe ascertained through the use of a ratio (e.g., user specified orautomatically assigned), then a relatively high confidence can beassociated with linking the text, and in particular the Java mention, toJava the computer programming language. The same can be done for othermentions in the text T, for example “computer programmer” and “school.”

As described previously, there are many techniques for assigningprobabilities to a string for given language models. One class oftechniques draws from ideas from the data compression literature. In thedata compression, the goal is to produce the shortest description for apiece of data so that it can be communicated efficiently to a receiver.When the sender and receiver share some common context (i.e. they aretold the text is being produced by a human thinking about a givenconcept), then the descriptions can be potentially further shortened bytaking advantage of the common context.

A basic result from the data compression literature is that the problemof assigning probabilities to strings of data is in many ways similar tothe problem of finding efficient representations of the text, and thusdata modeling techniques have an implication on data compressionalgorithms (through the use of algorithms such as arithmetic coding) andvice versa, algorithms for data compression can be used to buildestimates of probabilities of strings.

The latter observation makes it possible to use data compressionalgorithms to perform many of the operations described above for thetask of estimating probabilities of strings. For example, suppose thatone wants to estimate the probability that the text:

T=“The computer programmer learned how to write Java in school” camewhile thinking of the concept “Java, the computer programming language.”What one could then do is take a description of Java, the computerprogramming language, and then compress the text T assuming both senderand receiver have access to this description; this can be done, forexample, by assuming that T is an extension of the description of theconcept and using a pattern matching compression algorithm such asLempel-Ziv 77 or 78 starting from the first character of T. One couldalso compress T using a general context. These compressions as describedabove result in probability estimates that can be used as describedabove.

Other methods of computing an estimate of the probability of the textgiven a concept or a general language model can be implemented byembodiments, and embodiments are not limited to the ways to perform theanalysis described herein. For example, one could extract X featuresfrom the text (each feature could be a subsection of the text) and thenthe analysis above can be applied described above for these features,instead of the entire text at the same time. Then instead of having, forexample, one quantity, one has X quantities. One way to aggregate theseX quantities is to multiply them. One way to select these features is tochoose as a feature a maximal sequence of words in T that also appearsin a data source for a concept. Another possibility is to choose amaximal sequence of characters in T that also appears in a data sourcefor a concept.

Turning now to FIG. 2, a process for automatically linking text toconcepts in a knowledge base is generally shown in accordance with anembodiment. In an embodiment, the linking is performed usingdifferential analysis. At block 202, a text string is received. The textstring can be a collection of words, a sentence, a paragraph, or a wholedocument. At block 204, data sources are selected based on contents ofthe text string. Each data source can correspond to a concept in theknowledge base and can be used to build language models. Each of thedata sources can include one or more collections of names for acorresponding concept, a description for the corresponding concept,sentences referring to the corresponding concept and/or paragraphsreferring to the corresponding concept. At block 206, probabilities thatthe text string is associated with each of the language models as wellas a generic language model are calculated. In an embodiment, thegeneric language model is derived from a generic data source notspecified to any of the concepts in the knowledge base.

At block 208, the text string is associated (e.g., via a link) to aconcept in a knowledge base based on a comparison of the probabilitiesas described herein. The comparison can include calculating linkconfidence scores for each concept based on a differential analysis ofthe probabilities. The differential analysis can include comparing theprobability that the text string is output by a language model builtusing a data source to the probability that the text string is output bya generic language model. The differential can also include comparingthe probability that the text string is output by a language model builtusing a data source to a probability that the text string is output by alanguage model built using a competing data source. In an embodiment,the text string can be linked to additional concepts in the knowledgebase. In an embodiment, the link can apply to just a subset of the textstring, with the subset indicated in the link. The words in the subsetcan be consecutive or non-consecutive in the text string. In anembodiment, a link can be created from the text string to one of theconcepts in the knowledge base based on a link confidence score of theconcept being more than a threshold value away from a prescribedthreshold.

An key issue to be addressed can include is the ability to do conceptualsearching for things that are not already previously present in anexisting concept graph, or alternately, the ability to automatically addentries in to an existing knowledge base. A process of accomplishingthis will now be described in accordance with an embodiment.

The process described herein can assume that a string of text is givenwhich represents the name of something that is to be linked conceptuallyto other concepts in an existing knowledge base. It can also be assumedthat there is a reference corpus of data that is given to the systemwhich at a minimum mentions this string of text from generallyinformative language. It is does not have to be assumed that this stringof text is presented in only one specific sense (in case the string oftext has multiple distinct senses), nor that this corpus of data isstructured as an encyclopedia or dictionary (both assumptions that wouldmake the task in question easier).

It is further assumed that there is an existing concept graph wherenodes of the graph represent concepts and edges between nodes representknown associations between the concepts. Each concept in the conceptgraph is given a name or collection of different names. There are noassumptions made in terms of whether the string of text refers to aconcept already defined in the concept graph or not. The concept graphand the reference corpus of data may or may not be related. Examples ofa variety of combinations of these scenarios will now be described.

In the first example, the corpus of data and the concept graph areentirely unrelated. For example, it is assumed that the reference corpusof data is the external web pages of IBM Researchers(http://researcher.ibm.com) which are a collection of web pages in whichresearchers describe their research interests, education, projects, listtheir papers, etc. For the concept graph, it is assumed that Wikipediais used as the knowledge base. Wikipedia articles may be assigned tonodes in the concept graph, and the edges between nodes may behyperlinks referencing a Wikipedia article from another Wikipediaarticle.

As an example, assume that the string of text is “Mambo”. In the contextof the IBM Researchers corpus of data, “Mambo” is an IBM project thatrelates to computing processor simulations, unlike the sense of a musicor a dance as it may be more commonly known in a general context. It isassumed that this sense of Mambo (the IBM project) is not present in theconcept graph, and thus is an opportunity to demonstrate the embodimentsdescribed in here. As shown in FIG. 3, the term “Mambo” is entered inbox 302.

In an embodiment, text is extracted from the reference corpus which maycontain information describing aspects of the “Mambo”. One possible wayof accomplishing this is to do a text search for the word “Mambo” in thereference corpus and then extract text containing these mentions of theword “Mambo” from the corpus of reference. The text search can be doneusing a number of techniques known in the art. When doing this textsearch, variants of the word “Mambo” may be searched for as well, forexample “mambo” (the lower case version of “Mambo”) or “MAMBO” may bealso searched.

As shown in FIG. 3, the result of the search for “Mambo” on thereference corpus is illustrated. As illustrated, multiple documents 304in the corpus contain mentions of the word “Mambo”.

Following the text search, the text containing these mentions isanalyzed using a technique for extracting concepts from this text. Theembodiments described herein provide one such technique; however, itwill be understood that other techniques may be used as well. Theextracted concepts can be assumed to be already previously present inthe concept graph. In FIG. 3, the concepts that were extracted from thetext containing these mentions are shown at 306.

The extracted concepts can be used in at least two distinct ways,namely, for a system query and for concept graph expansion.

With respect to the system query use, the extracted concepts may beapplied as a query to a system that can perform conceptual searching orrecommendations. Such a system does not have to return documents relatedto the concept graph or the corpus of reference (although it could alsodo that). The key assumption though is that the system can accept as aninput query the concepts extracted in the second step above, and that itis capable of returning documents conceptually associated with the inputquery, independently of the origin of these documents.

With respect to the concept graph expansion use, a new node in theconcept graph is added with the name “mambo”, and is connected to othernodes in the concept graph using the extracted concepts. Additionally,the text from which the concepts were extracted, their provenance andthe confidence of the extracted concepts may also be added as additionalmetadata in the new concept definition.

In many situations the steps described above will be successful. But insome situations, the text containing the mentions of the string of textmay contain references to concepts that are unrelated to the potentialnew concept. This can problem can be remedied through the followingtechnique. The concepts extracted from the text containing the mentionscan undergo a process of “de-noising”. In the context of this problemde-noising refers to the act of estimating the cross-relevance of theconcepts extracted (possibly sometimes even removing them). If there isa subset of concepts extracted that have some degree of associationswith each other, then the confidence that those extracted concepts canbe used in the new concept definition increases. Alternately, if thereare extracted concepts for which other related extracted concepts cannotbe found, then the confidence that those extracted concepts can be usedreliably in the definition of the new concept diminishes.

Estimating the relation between any two concepts can be attained byexploiting a concept graph. In particular, the concept graph may beutilized to determine these relations (e.g., whether they appearexplicitly in the form of links in the concept graph between the twoconcepts, or implicitly through an exploration and weighting of thevarious paths that connect the concepts in the concept graph). Inanother embodiment, a technique is described for estimating how close isa specific concept in a collection of extracted concepts to the rest ofthe extracted concepts by analyzing the top M closest concepts and thenmerging those M scores using information combining techniques. A sampleof top concepts is shown generally at 308 in FIG. 3.

In some situations, it is possible that the reference corpus of data maycontain references to the same string of text that refer to two or moresenses. For example, one may postulate that the pages of the IBMresearchers may sometimes refer to “MAMBO”, a local club with interestson Latin-American music. A search for “Mambo” may (after lower/uppercasing normalization is applied) result in documents containing bothreferences to the project as well as references to the club. If there ismetadata that can be used to separate these two senses (for example,some pages are known to be about projects, and some pages are known tobe about hobbies), then the extracted concepts can be separated in twogroups (or more than two groups, depending on the situation). If no suchmetadata is available, or if the metadata is unreliable so that it needsto be aided by further analysis, embodiments of the invention mayutilize clustering algorithms to attempt to separate these twointermixed senses, as will now be described.

In an embodiment, it is assumed that a collection of extracted conceptsis provided. From these concepts, feature vectors are obtained. Onepossible feature vector is to compute the likelihood vector for eachconcept that indicates how likely it is that the given concept isrelated to all possible other concepts in the concept graph. Otherfeature vectors may subsample the feature vector described above. In thetwo examples above, the concept graph is used to compute those featurevectors. Once in possession of the feature vectors, clusteringalgorithms (for example k-means) may be employed to cluster theassociated concepts in groups. The clustering separates concepts intodistinct groups on the basis of their conceptual closeness. In theexample above, extracted concepts related to computer simulations,processors, etc. are clustered in one group and concepts related toLatin-American music and hobby clubs are clustered in another cluster.

Once in possession of two or more clusters, one can proceed to employthe results of this analysis. The two clusters are regarded as twodistinct senses of the same string of text. As stated before, there aretwo applications described so far: in the first application, the stringof text is interpreted as a system query in which documents are returnedfrom some corpus through conceptual associations. In the secondapplication, the goal is to augment a concept graph with a new conceptdefinition.

In the first application, because multiple senses of the given string ofword have been discovered, it is not possible to immediate perform aconceptual query. However there may be contextual information that canbe used to select a most likely sense (for example, it may be known thatthe general line of inquiry is around “computer science” due to theapplication at hand or background information about the user, or thebrowsing history of a user). In resonance with the embodiments describedherein, it is assumed that the contextual information can be summarizedas a collection of concepts from a concept graph. The concept graph maybe employed along with the various techniques described in thisembodiment to measure the closeness between the concepts in thecontextual information and each of the groups of concepts that define asense of the string of text. Once this closeness is established, themost likely sense can be computed. For example, if there is a sensewhich sufficiently separates with respect to the rest of the sensesgiven the contextual information, then this sense is selected as theintended sense and a conceptual query is performed for the sense.Alternately, a user is presented with the option of selecting whichsense of the string of text was meant, via a dialog on a user interface.Once the sense has been disambiguated by the user, the conceptual querycan be performed as indicated above.

In the second application, which is that of enlarging a concept graph,the various senses of the string of text may be added separately to theconcept graph as described above.

In the application where one seeks to enlarge a concept graph, animportant challenge is that of automatically extracting strings of textwhich may be candidates for new concepts. One technique for automaticextraction can include processing the reference corpus using naturallanguage processing technology and extracting all noun phrases that itcontains. These noun phrases are natural candidates for incorporation asnew concepts into the concept graph. The pool of noun phrases is thenordered according to some criterion of importance. Generally speaking,noun phrases that appear more frequently can be given higher priority,as there is evidence they are discussed with certain frequency, and alsothe quality of the definitions that can be derived is higher.

One challenge involves deciding when the task of automatically defininga concept is moot because the underlying concept is already present in aconcept graph. This may be resolved by comparing the names of theconcepts in a concept graph with the string of text, and when finding asufficiently close match, then computing the conceptual closenessbetween the concepts extracted in the process of defining the string oftext (for a single sense, in case where there are multiple senses foundfor the string of text in the reference corpus) and the concepts towhich the close match concept found earlier is linked to in the conceptgraph. If there is a sufficiently high correlation between these twosets of concepts, then it is determined that it is not necessary todefine a new concept, and instead the original concept in the conceptgraph may be sufficient.

One embodiment provides the capability to determine the meaning (ormeanings) of a hashtag presented in a user's social media application(e.g., TWITTER). Typically, it is very difficult for a computer tounderstand the meaning of a hashtag from a single tweet because of thelength of the tweet and the lack of context it has about the tweet.However, if one regards a large collection of tweets as the referencecorpus, then analysis of this larger amount of text has a higherprobability of successfully being able to define the meaning (ormeanings, in case it has multiple senses) because analysis of thetweets' text using the embodiments described herein will result inlinkages into an existing concept graph with higher probability than ifa single tweet was employed. Here, the concept graph could be derivedusing Wikipedia, freebase, or other data sources.

The embodiments described herein offer great flexibility. In the earlierexample, the reference corpus and the concept graph were not related.However, there are instances where the reference corpus and the conceptgraph may be closely related. For example, the concept graph may bederived from the articles of Wikipedia (e.g., where each node of theconcept graph would have a name given by a Wikipedia article and itslinks to other nodes would be determined by the hyperlinks connectionarticles in Wikipedia). The reference corpus could be Wikipedia itself,including the full texts of each Wikipedia article. In this case, notethat there is a very large number of noun phrases in Wikipedia that arenot present as Wikipedia article names. These can be prime candidatesfor new concepts in a concept graph, demonstrating that the embodimentsare also applicable and meaningful when the concept graph and thereference corpus are closely related to each other.

Turning now to FIG. 4, a process for automatic new concept definition isgenerally shown in accordance with an embodiment. At block 402, a stringof text is received, and at block 404, a corpus of data can be searchedto locate additional text related to the string of text. The searchingcan include searching for both the string of text and for variants ofthe string of text. At block 406, concepts can be extracted from theadditional text. The extracted concepts can include a subset of conceptsin a concept graph and can be de-noised to estimate the cross-relevanceof the extracted concepts. In an embodiment, the corpus of data and theconcept graph are separate entities. Processing then continues at block408 or block 410.

At block 410, the extracted concepts can be used to link the string oftext to the concept graph. It is determined whether the string of textshould be linked to an existing concept in the concept graph. Thedetermining can include determining similarities between the string oftext and a name of the existing concept, and then deciding whether thestring of text should be linked to the existing concept based on thesimilarities. The determining can also include determining a conceptualcloseness between the extracted concepts and the concepts to which theexisting concept is linked to in the concept graph, and then decidingwhether the string of text should be linked to the existing conceptbased on the conceptual closeness. The linking can be performed if itwas determined that the string of text should be linked to the existingconcept in the concept graph. Alternatively, a new concept that isassociated with the string of text can be added to the concept graph ifit was determined that the strong of text should not be linked to theexisting concept in the concept graph. Adding the new concept to theconcept graph can include linking the new concept to at least one of theextracted concepts from the additional text. In an embodiment, theextracted concepts can be clustered into one or more clusters withrelated concepts. Based on their being two or more clusters, thefollowing can be performed independently for each cluster: determiningwhether a new concept should be added to the concept graph based on thesimilarities of the string of text to the names of existing concepts andthe similarities between the concepts in the cluster to the conceptslinked to an existing concept; and based on determining that a newconcept should be added, adding the new concept to the concept graph.

Referring back to FIG. 4, at block 408, the extracted concepts are usedin a conceptual query, and at block 412 documents that are conceptuallyrelated to the extracted concepts are returned based on the query.

An embodiment includes computing the relevance of a document to conceptsnot specified in the document. Techniques described herein can be usedfor performing conceptual document integration techniques to create adeep vector representation of a document.

Embodiments disclosed herein relate to the use of a concept graph todefine a space of concepts and their relations. As described previously,a concept graph can be derived from crowd-sourced data sources such aswikis which focus on defining concepts (e.g., Wikipedia) and canadditionally be augmented with concepts found in new unstructured datasources. In an embodiment, concepts that exist in a concept graph areextracted from a document and then a vector representation of thedocument is created in “concept space” in which every dimension is aconcept of the concept graph. As used herein, a concept space is acollection of two entities. One of the entities is a vector space with Ndimensions. The other entity is a concept graph with N concepts, givinga precise meaning to each of the dimensions of the vector space. Theextracting of concepts from a document can be done using “wikification”techniques which are known in the art. Unlike the term vector spacemodel that is commonly used in the information retrieval community,where a document is represented by the frequencies of its terms (oftenalso including the inverse document frequency of the terms), the vectorrepresentation described herein can have a value assigned to a vectorentry that represents the relevance of the concept to the document orvice versa, and thus it has a much higher level semantic significance.It is important to note that in the concept vector representation of adocument, the document may have nonzero scores for concepts that areactually not present in the document, but that are inferred to berelevant to the document using reasoning afforded to embodiments by theconcept graph.

In what follows, reference is made to raw data that has been extractedfrom a document using for example, text annotation techniques such asthe ones described previously which have the role of picking outmentions of concepts in the text. Reference is also made to inferencesthat are made from this data using a concept graph as an aid to makethese inferences. The raw data extracted above is referred to herein asa priori data, or information, about the document. The inferences thatare made on this data with the goal of computing a relevance score forthat document for each possible concept in a concept graph are referredto herein as a posteriori inferences (or a posteriori information). Thea priori and a posteriori labels refer to the data about a documentbefore and after the application of knowledge from the concept graph,respectively.

As an example, suppose that from a document D concepts c1, c2, c3, . . ., cK have been extracted. Together with these concepts, confidencescores s1, s2, s3, . . . sK have also have been extracted. Theseconcepts c1, c2, c3, . . . cK together with the confidence scores s1,s2, s3, . . . sK constitute what is referred to herein as a prioriinformation about the document. A confidence score can be a measurementor estimate of how sure the correct concept was extracted by the conceptextraction process. In accordance with an embodiment, the goal is toobtain the relation between the document D and a general space ofconcepts (e.g., all or a subset of a concept graph). This representationcan connect the document D with concepts not necessarily present in theoriginal description of it, via exploitation of deep conceptualconnections as seen in a concept graph (hence the name “deep conceptvector representation”). This representation is referred to herein asincluding a posteriori inferences about the document which have nowincorporated additional knowledge as related to concept. One type ofrelation to obtain is to compute how relevant a document would be for aquery comprised of one concept.

An embodiment includes taking a priori data about a document and thenemploying a concept graph to improve the conceptual understanding ofthat document. Note that the concepts c1, c2, c3, . . . ck may have comewith confidence scores s1, s2, . . . sk from the annotation techniquesassociated with the concept extraction. In a first step, theseconfidence scores can be further refined by taking into account therelations between the scores as made available through computations overa concept graph.

For the sake of an example, assume that the concepts extracted from adocument are given by the list [“vanilla yogurt”, “superman”,“superman”, “superman”, “the green lantern”, “batman and robin”]. Tosimplify it is assumed that the confidence scores of each of theseannotations is 0.7. An observation by an informed human of this listcould lead to the conclusion that this document is probably about superheroes in comics, with some emphasis on superman. The concept “vanillayogurt” appears to be incongruent to the rest of the list, and there isno evidence to give it a strong weight, but similarly, no evidence toeliminate it from the list. Note that if the list had been [“the greenlantern”, “vanilla yogurt”] there would have been no reason to concludethat this document was particularly about comics, any more that it wasabout desserts, since the concept “vanilla yogurt” would still requireexplanation. The point of this example is that the various comics heroesreferences tend to reinforce each other's presence in the document D andthe general notion that the document D is related to comics, and inparticular DC Comics, grows as more comic related references are addedto the list. In a computing system, the information about theconnections between all of these concepts in the list comes from theconcept graph. Incorporation of the knowledge in the concept graph, maylead to the refinement including increasing the confidence scores foreach instance of “superman” to, say, 0.85 (from 0.7), to increasing the“the green lantern” and “batman and robin” confidence scores to 0.8(from 0.7), and possibly to downgrade the “vanilla yogurt” score to 0.65(from 0.7). These inferences are no longer a prior inferences, howeverthis only one possible first step towards the general goal of conceptualunderstanding of the document.

Continuing with the example, suppose that the items [“wonder woman”,“aquaman”] are added to the list [“vanilla yogurt”, “superman”,“superman”, “superman”, “the green lantern”, “batman and robin”]. Atthis point, it is not particularly surprising that these kinds of thingsare being added to this list. It is still useful information that can bereflected in the final representation of the document, but at this pointadding [“wonder woman”,“aquaman”] to this list may not have as muchvalue as adding [“the green lantern”, “batman and robin”] to the list[“vanilla yogurt”, “superman”, “superman”, “superman”]. This exampleillustrates that a proper conceptual summarization is likely to accountnot only for the reinforcement between concepts, but also for theverbosity in a description. For purposes of illustration, the confidencescores after “wonder woman” and “aquaman” are added may only increasethe confidence for “superman” to 0.9 (from 0.85).

With these two motivating examples in mind, a formalism for identifyingthe updated confidence scores w1, w2, w3, . . . , wK is now described inaccordance with an embodiment. Assume that the concept space has a totalof N concepts, where typically N is much greater than the number ofconcepts ever present in a document of reasonable size. A concept vectorrepresentation of the document D will be denoted as r(D) and willcomprise N numbers, each describing the likelihood that the document Dis related to each respective concept, or even more particularly, howrelevant the document D is to a single concept query. An individualconcept can be regarded as a document with a single extracted concept,and thus via a slight notation overload, one can denote by r(ci) thevector with N entries that describe the likelihood that the concept ciis related to any other concept in the concept space. Computing thevector r(ci) can be done with a variety of techniques, including onesbased on Markov chain simulations.

Recalling the example given before, a process can be performed to learnhow each of the concepts c1, . . . , ck are related to each other. Letr(ci)[ck] denote the likelihood that concept ci is related to ck, andassume that r(ci)[cj]=½ if there is no relation whatsoever. The matrix:

$\begin{matrix}{{r\left( {c\; 1} \right)}\left\lbrack {c\; 1} \right\rbrack} & {{r\left( {c\; 1} \right)}\left\lbrack {c\; 2} \right\rbrack} & {{r\left( {c\; 1} \right)}\left\lbrack {c\; 3} \right\rbrack} & \ldots & {{r\left( {c\; 1} \right)}\lbrack{cK}\rbrack} \\{{r\left( {c\; 2} \right)}\left\lbrack {c\; 1} \right\rbrack} & {{r\left( {c\; 2} \right)}\left\lbrack {c\; 2} \right\rbrack} & {{r\left( {c\; 2} \right)}\left\lbrack {c\; 3} \right\rbrack} & \ldots & {{r\left( {c\; 2} \right)}\lbrack{cK}\rbrack} \\\ldots & \; & \; & \; & \; \\{{r({cK})}\left\lbrack {c\; 1} \right\rbrack} & {{r({cK})}\left\lbrack {c\; 2} \right\rbrack} & {{r({cK})}\left\lbrack {c\; 3} \right\rbrack} & \ldots & {{r({cK})}\lbrack{cK}\rbrack}\end{matrix}\quad$contains the essential information needed to determine how conceptsrelate to each other and how verbose a description is, among otherproperties of analytic interest.

Recall that for a likelihood p the likelihood ratio (LR) can be definedas LR(p)=p/(1−p), and that the log likelihood ratio (LLR) can be writtenas LLR(p)=log(p/(1−p)). Also call LR^{-1}(x) and LLR^{-1}(x) thecorresponding inverse functions. Choose a column, for example the ithcolumn. By examining the column of this matrix, a number of statisticscan be deduced, including an estimate of how many other concepts arerelated to concept ci in one way or another. In general, the row vectorv=[sum_{i=1}^{k}f(r(ci)[c1],si)sum_{i=1}^{k}f(r(ci)[c2],si) . . .sum_{i=1}^{k}f(r(ci)[cK],si)]  (1)where f( ) is a function that defines how to count the contribution of amatrix entry, can be seen as a general statistic that can be computed byspecializing the function f(.). In the example above, f(x,s)=0 if x=½and f(x,s)=1 if x>½ would be one possible choice. Another possiblechoice that actually uses the confidence scores would be f(x,s)=0 if x=½and f(x,$)=s if x>½. Other choices for f are possible depending on theapplication. For convenience, individual elements of the vector v arereferred to as v(cj)=sum_{i=1}^{K} f(r(ci)[cj], si).

The row vector thus computed is an estimate for how much the generalarea of a concept is being mentioned, taking into account informationfrom all other extracted concepts. A large value in an entry of the rowvector implies a larger presence of the underlying concept, andvice-versa. The updated confidence scores w1, w2, w3, . . . , wK on theconcepts can be defined as a function of the corresponding elements ofv:wj=g(v(cj)).  (2)This function g(.) is an application dependent quantity that that can beused to control how much concept reinforcement is promoted and how muchconcept verbosity is promoted/demoted. Choices for g( ) are describedherein below.

A unified a posteriori view of the document is formed. As describedearlier, it can be assumed that from a document D concepts c1, c2, c3, .. . , cK have been extracted (possibly with confidence scores s1, . . .sK) and that via a procedure like the one described above, updatedconfidence scores w1, w2, w3, . . . , wK are derived for each of theextracted concepts. The task is now to create a view of how the documentD relates to each of the N concepts in the concept space. Recall thatthe vector r(ci) describes the likelihood that the concept ci relates toevery one of the N concepts. Thus, the document D can be mapped onto theconcept space by weighting the vectors r(ci) using the updatedconfidence scores w1, w2, . . . wk:sum_{i=1}^{K}wi LLR(r(ci)))  (3)where the above can be seen as a weighted average of log likelihoodratios, and where LLR(x) where x is a vector is obtained by applying theLLR function to each entry of the vector (with a similar statementholding for LLR^{−1}). To map this number back to a probability space,computeLLR^{−1}(sum_{i=1}^{K}wi LLR(r(ci)))  (4)

The function g(.) that is used in the equation (2) above to compute theweights can be defined as follows. Suppose that g(x) is set to equal1/x, and suppose that f(.) in equation (1) above is set to be f(x,s)=0if x=0 and f(x,s)=1 if x>0. Then, the vector v is simply representinghow many concepts exist in the vicinity of a concept (disregarding theinitial confidence score), however tenuous the connection may be. Inprevious example, [“vanilla yogurt”, “superman”, “superman”, “superman”,“the green lantern”, “batman and robin”, “wonder woman”, “aquaman”] thean assignment for the values in vector v can be:

v(“vanilla yogurt”)=1

v(“superman”)=7

v(“the green lantern”)=7

v(“batman and robin”)=7

v(“wonder woman”)=7

v(“aquaman”)=7

The v(“superman”)=7 is derived from the fact that there are 7 conceptsin that list that are at least moderately related superman. Similarstatements can be made for the rest of the concepts above.

As a consequence, the choice g(x)=1/x in equation (3) above iseffectively uniformly weighting clusters of concepts extracted from thedocument, no matter how many different concepts there are in thecluster. Specifically, and using the example above, the document can beintegrated using the computation:

r(“vanilla yogurt”)+ 1/7 r(“superman”)+ 1/7 r(“superman”)+ 1/7r(“superman”)+ 1/7 r(“the green lantern”)+ 1/7 r(“batman and robin”)+1/7 r(“wonder woman”)+ 1/7 r(“aquaman”)

The embodiment of the assignment above does not account for conceptreinforcement and verbosity as described earlier. This is because theentire set of 7 comic related concepts, given that they are each beingweighed down by a factor of 7 do not contribute as an ensemble any morethan the “vanilla yogurt” concept does. It may be desirable to increasethe 1/7 weight to some other larger number. This discussion is purelymeant to demonstrate that a variety of different behaviors are possiblewithin the scope of embodiment. In the following text, a technique isdescribed that accounts for concept reinforcement and verbosity inaccordance with an embodiment.

In one embodiment, g(x) is assigned to equal h(x)/x for some functionh(x) that is growing with x, and that satisfies (for example) h(1)=1.That way the factor in front of r(“vanilla yogurt”) remains 1.0, whilethe factors in front of the other concepts will be larger than currentlyassigned. The function h(x) then is set to grow slower than linearlywith x, for example, h(x)=x^u for some u 0<u<1.0. This general class offunctions falls within the scope of our embodiments and also account forreinforcement and verbosity. The reason for the slower than lineargrowth for h(x) is that assigning h(x)=x (hence g(x)=1) also defeats theinitial purpose as then the weights become r(“vanillayogurt”)+r(“superman”)+r(“superman”)+r(“superman”)+r(“the greenlantern”)+r(“batman and robin”)+r(“wonder woman”)+r(“aquaman”) and thenthe comics related concepts effect in the overall unified concept spaceview of the document is unreasonably large.

The computation shown above in Equation (4) is performed for someweights w1 . . . wk that are derived in ways as described above. Thiscomputation is a vector with N entries, since each of the r(ci) is alsoa vector with N entries. Recall that this vector encodes the likelihoodthat a concept ci is related to each other concepts in a concept graph.Additionally, note that the concept graph was used both in thecomputation of r(ci) (via the Markov chain techniques described earlier)as well as in the computation of the updated confidence scores wi. Ingeneral, the concept graph can be used to improve one but not the other;however the example above includes both for completeness. The result ofEquation (4) is the a posteriori likelihood inferences sought byembodiments, as it connects the document to all other concepts in aconcept graph by processing the a priori data with the concept graphinformation.

An alternative embodiment can be utilized for processing the a prioriinformation about the document and producing the a posteriorilikelihoods relating a document other concepts. An advantage to thisembodiment is that it can allow processing of an arbitrary large set ofconcepts in a document with a finite amount of memory in the processingsystem. Another advantage is that it is able to differentiate in thefinal a posteriori likelihood vector the contributions on the conceptsexplicitly present in the document as per the initial conceptextraction. The fundamental idea is to choose, for every concept1<=j<=N, the top M extracted concepts closest to the concept j. Then,the likelihood that each of these top M concepts are related to theconcept j is combined using information combining mechanisms. Findingthese top M concepts can be done by processing the concepts c1, . . . ,ck sequentially. The second key idea is that in the output likelihoodvector, the K concepts c1, . . . ck receive a special treatment beyondthe above. This is important so as to ensure that document queries thatcontain an extracted concept that is also a query are rankedappropriately.

In the alternative embodiment, it is assumed assume that there is apreprocessing stage that eliminates repeated concepts and keeps countsn1, . . . , nk that reflect how many times these concepts repeat and theconfidence scores with which they appear. Furthermore r(ci) is modifiedwith a function that is monotonic with these counts and scores. For thesake of the description of this embodiment, concepts with theirrespective index are identified in the vector for N concepts. So, inparticular ci is an integer for 1<=i<=k. For each concept ci in {c1, . .. , ck} the contribution to the a posteriori likelihood for each of theN concepts in the concept graph can be computed as follows. Suppose thatj is one of the N concepts in the concept graph (resulting in j \in \{1,. . . ,N\}). If ci==j then a “direct hit”, then contribution to the aposteriori likelihood of c1 is computed asdirect_{j}=lambda_d r(j)[j]+gamma_dIf ci !=j, then a “side” contribution to the a posteriori likelihood ofj is computed as follows. The contribution is given byy_{i,j}=lambda_s r(ci)[j]+gamma_s

The compounded contribution of the “side” hits is computed by selectingthe highest M among the list [y_{1,j}, y_{2, j}, . . . , y_{k,j}1,computing their likelihood ratios, multiplying them, and then computingthe likelihood value side] from the resulting likelihood ratio.

In the case that concept j does not appear in {c1, . . . ck}, the finala posteriori likelihood for concept j can be given by:z_j=a_side side_j+b_side

In the case that concept j DOES appear in {c1, . . . ck} then the finala posteriori likelihood for c1 can be given by:LR^{−1}(LR(z_j)LR(direct_j))

In additional embodiments, lambda_d, gamma_d, a_side, b_side can varywith i as a function of the popularity of the concept ci.

Turning now to FIG. 5, a process for computing the relevance of adocument to concepts not specified in the document is generally shown inaccordance with an embodiment. At block 502, a concept graph isaccessed. At blocks 504 and 506, a relevance of the document to conceptsin the concept graph is computed. At block 504, a priori informationabout a document is received. The a priori information can includeconcepts previously extracted from the document. The a prioriinformation can also include confidence scores corresponding to each ofthe concepts extracted from the document.

At block 506, a posteriori information about the document is generatedthat indicates the likelihood that the document is related to each ofthe concepts in the concept graph. The a posteriori information can begenerated by combining the a priori information and the concept graph.The a posteriori information can be responsive to paths connecting eachof the concepts in the concept graph to each concept in a selectedsubset of the concepts extracted from the document. The selected subsetof the concepts extracted from the document can vary based on theconcept in the concept graph and can include a specified number of theconcepts extracted from the document that are the most related to theconcept in the concept graph. The a posteriori information can include aweighted averaging of vectors associated with extracted concepts. Theweight for an extracted concept can be responsive to a degree to whichthe extracted concept is related to the other extracted concepts and/orto a frequency in which the extracted concept appears in the document.

A relevance of the document to concepts not extracted from the documentcan also be computed. The document can be in a corpus of documents andthe processing can further include utilizing the a posterioriinformation to search the corpus of documents. The process can alsoinclude outputting a threshold number of concepts from the concept graphhaving the highest likelihood, the output based on the results of thecombining.

Embodiments described herein further include the use of efficient datastructures for storing and querying deep conceptual indices. Asdescribed previously, a document can be received, concepts extractedfrom the document, confidence scores calculated for the various concepts(with the goal of, for example, promoting concept reinforcement andcreating diminishing returns by penalizing exceedingly verbosedescriptions), and then a representation of the document is created in aconcept space that connects the document to all possible concepts (notonly the ones that were found explicitly in the document). Embodimentsdescribed herein are directed to how this information can be organizedin a computer system so that it can be efficiently queried against andmaintained. A query can typically be formed by specifying a concept or aset of concepts in a user interface directly or indirectly by stating aquery in natural language from which concepts are then extracted.

In embodiments one or more concepts can be selected as the query. Forease of description herein, it is assumed that a single concept q hasbeen selected as the query. The computational task involved in anembodiment is that of examining the score that has been assigned toevery document for the query q, and returning the documents in the orderimplied by that score. This can be achieved by building an invertedtable that maintains for every concept, a list of documents andassociated their score. This allows for quick retrieval of this listupon receiving the query concept. The inverted table utilized byembodiments described herein is of a very different nature than invertedtables used in traditional information retrieval mechanisms, where forevery keyword (or generally some text surface form) a list of documentscontaining that keyword is listed. Some fundamental differences includethat instead of keywords, the abstract notion of a concept is used,which is defined in embodiments as a node in a conceptual graph; andthat a document is associated with a variety of concepts that are notpresent in the document to start with. The list of documents can bemaintained in sorted form (e.g., sorted by score) and can be organizedas a hash table as well, where they key is the document and the value isthe score.

In an embodiment of the conceptual indexing technique described herein,a document is associated with potentially a large number of concepts,and hence it can become important to design and use data structures thatare capable of very fast insertion, deletion and updating of scores. Inparticular, in order to be able to very quickly invalidate previousscores of a document, the notion of conceptual index versioning can beintroduced. Scores can be augmented with a version number, and anadditional metadata structure can be added to tracks what is the mostrecent version of a document. Incrementing the version number results inautomatically and immediately invalidating all the scores of a documentin the inverted table with a version number lesser than the new versionnumber.

The conceptual index versioning system described above can also allow aseamless experience in transitioning scores for a document to newscores. This can be accomplished as follows: the new scores for thedocument are uploaded with the new version number, however the versionnumber for the document indicated in the metadata is not updated untilall the scores for the new version have been added to the invertedtable. The inverted hash table data structure can also be augmented witha garbage collection mechanism which periodically deletes concept scoresof versions that are no longer valid in the system. In addition, thegarbage collection mechanism can delete scores (even if current,matching the existing version) starting from the lowest scores, wheneverthere is existing or predicted space pressure in the table.

An example of the indexing follows. Assume that a conceptual index needsto be created for the following two documents:

D1=“Vanilla ice cream is boring” and

D2=“Induction cooking is all the rage these days.”

After extracting concepts from D1, “Vanilla ice cream” is obtained andafter extracting concepts from D2, “Induction cooking” is obtained.

Now suppose, for the sake of the discussion, that the space of possibleconcepts that are being considered is:

[“Vanilla ice cream”, “Induction cooking”, “Kitchen”, “VanillaPlanifolia”, “Dairy”, “LCD screen”].

Note that in practice, a concept space is typically much larger andoften in the order of millions to hundreds of millions of concepts. Now,using the techniques described previously, scores can be determined forD1 and D2 for the space of concepts. For example:

-   -   Scores for D1 can equal 1 {“Vanilla ice cream”: 1.0},        {“Induction cooking”: 0.2}, {“Kitchen”: 0.3 }, {“Vanilla        Planifolia”: 0.9}, {“Dairy”: 0.9}, {“LCD screen”: 0.001}}; and    -   Scores for D2 can equal 1 {“Vanilla ice cream”: 0.1},        {“Induction cooking”: 1.0}, {“Kitchen”: 0.8}, {“Vanilla        Planifolia”: 0.01}, {“Dairy”: 0.15}, {“LCD screen”: 0.1} 1.

In this example, the conceptual inverted table would read:

“Vanilla ice cream”: (D1, 1.0, v1) (D2,0.1,v1)

“Induction cooking”: (D2, 1.0, v1) (D1, 0.2, v1)

“Kitchen”: (D2, 0.8, v1) (D1, 0.3, v1)

Vanilla Planifolia”: (D1, 0.9, v1) (D2,0.01,v1) “

Dairy”: (D1, 0.9, v1) (D2,0.15,v1)

“LCD Screen”: (D2, 0.1, v1) (D2,0.001,v1)

In a separate metadata structure, the current version number for thescores of the documents can be indicated as {“D1”: v1, “D2”: v1}.

Now suppose that D2 is updated to read “Induction cooking is all therage these days!! In my kitchen remodeling, I additionally plan to havea fridge with an LCD screen.” Then the new scores for D2 may become:

-   -   Scores for D2={{“Vanilla ice cream”: 0.1}, {“Induction cooking”:        1.0}, {“Kitchen”: 1.0}, {“Vanilla Planifolia”: 0.01}, {“Dairy”:        0.151, {“LCD screen”: 1.0}}

Then, the new scores can be inserted into the conceptual inverted tableresulting in:

“Vanilla ice cream”: (D1, 1.0, v1) (D2,0.1,v1) (D2,0.1,v2)

“Induction cooking”: (D2,1.0,v2) (D2, 1.0, v1) (D1, 0.2, v1)

“Kitchen”: (D2,1.0,v2) (D2, 0.8, v1) (D1, 0.3, v1)

“Vanilla Planifolia”: (D1, 0.9, v1) (D2,0.01,v1) (D2,0.01,v2)

“Dairy”: (D1, 0.9, v1) (D2,0.15,v1) (D2,0.01,v2)

“LCD Screen”: (D2,1.0,v2) (D2, 0.1, v1) (D2,0.001,v1)

Note that in the above the list is kept in ordered form, however inother embodiments, the list is not kept in ordered form.

After the update above happens, then the version of the documents areupdated to reflect {“D1”: v1, “D2”: v2}.

When accessing the inverted table below, the document scores for D2 thatdo not match v2 can be discarded. A garbage collection mechanism canthen scan the conceptual inverted table above and delete entries thatare no longer current, leaving:

“Vanilla ice cream”: (D1, 1.0, v1) (D2,0.1,v2)

“Induction cooking”: (D2, 1.0, v2) (D1, 0.2, v1)

“Kitchen”: (D2, 1.0, v2) (D1, 0.3, v1)

“Vanilla Planifolia”: (D1, 0.9, v1) (D2,0.01,v2)

“Dairy”: (D1, 0.9, v1) (D2,0.01,v2)

“LCD Screen”: (D2, 1.0,v2) (D2,0.001,v1)

Embodiments can be utilized for queries that include multiple concepts.Multiple concept queries can be present when a user explicitly (orimplicitly, via the use of natural language) describes two or moreconcepts with the hope of retrieving documents that are relevant to areasonable fraction of the given concepts. Another multiple conceptquery scenario is when a user introduces a *document* as an examplequery, and in this case the concepts that are included in the documentcan be used to formulate a multiple concept query. Yet another scenariois when documents and concepts are mixed as a query.

The following text describes several approaches to using the conceptualindexing system described above in the context of multiple conceptqueries. A common thread throughout these approaches is that an Mconcept query (where M is some positive integer) will result in Mindependent lookups in the conceptual inverted table. There are severalways to combine the results of such lookups. One way to interpret arequest for an M concept query is that the requestor wants to obtain aranking of the documents which promotes documents that would have shownup with high confidence scores independently in the list of results ofthe separate M queries for the M individual concepts.

Mathematically, if there is a list of documents [D1, D2, D3, . . . ,DL], and as a result of a query for a single concept ci (i=1 . . . M)the scores [s_{i,1} s_{i,2} s_{i,3} . . . s_{i,L}] are obtained, one canobtain a unified scoring for a query for all M concepts by an averaging(straight or weighted) of all the M row vectors above. In some casesnonetheless, the information retrieval system may not be fullycalibrated and as a result the various vectors in [s_{i,1} s_{i,2}s_{i,3} . . . s_{i,L} ] may not be compatible with each other. Forexample, a score of 0.8 for a query on “information retrieval” may implya different relevance than a score of 0.8 for a query on “FederalHousing Administration”, even for the same document. As a result, inthese instances an averaging, whether weighted or not, of the vectors in[s_{i,1} s_{i,2} s_{i,3} . . . s_{i,L} ] can lead to undesirable orunexpected outcomes in the combined ranking. One technique for improvingthis potential problem is to compute, for every i=1 . . . M, the rank ofelements of [s_{i,1} s_{i,2} s_{i,3} . . . s_{i,L} ] after sorting, andthen to average the ranks themselves, instead of the scores directly.

An example follows for the purposes of clarifying the computationtechnique for the combined scores. Suppose that a combined search isbeing performed for two concepts in L=4 documents D1, D2, D3, D4, andsuppose that the scores obtained are concept1 [0.4 0.8 0.0 1.0] concept2[0.7 0.1 1.0 0.3] respectively. By sorting the scores from highest tolowest for each query and noting the resulting index, concept1 [3 2 4 1]concept2 [2 4 1 3] is obtained.

If one were to use an averaging of the rows in [s_{i,1} s_{i,2} s_{i,3}. . . s_{i,L} ] directly then one would obtain for the combined scores[0.55 0.5 0.5 0.65] where higher is better. If the ranks were usedinstead, then averaging the ranks would result in [2.5 3 2.5 2] wherelower is better.

In some situations, it is desirable to find documents that are relatedto other documents. This is referred to as a query in which adocument-by-example is provided. These queries are nothing other thanconceptual queries, since the document that is being used as a query canbe analyzed, extracting one or more concepts from this document andusing those concepts in the query. In an embodiment, for a multipleconcept query with document-by-example, the document is first analyzedto extract a list of concepts for the document that are deemed to behighly relevant, and then using the techniques previously described canbe used to perform a multiple concept query. Embodiments deduce theidentity of the important concepts in a document by restricting theconcepts to be considered to be only those initially extracted from thedocument. Nonetheless, the entire concept graph is used to analyze howthese concepts relate/reinforce each other, using the techniquesdescribed previously herein. The results of that scoring are then usedin order to select the top concepts to be used in the multiple conceptquery.

Because the fundamental entity in the query is a concept, and documentsin the document-by-example are simply converted into a one or moreconcept query, it is also possible to create queries that combineconcepts and documents simultaneously, as shown in FIGS. 13B and 13Cbelow. Suppose for example that the query has exactly one document andone concept being specified. In the example of an expertise locator, itmay be the name of a person, and an additional concept to furthersharpen the query. Suppose the person is associated with a document,which in turn is associated with 50 concepts. The top C (e.g., C=10)concepts for that person are selected and then a document-by-examplequery is made using the person as an example. This results in a list ofdocuments from the system. In addition, the single concept query is madeseparately, resulting in a second list. Finally, these lists are joinedusing some criterion. The criterion described above which combinesvarious lists by averaging the ranks of the items in the list may beused as a joining algorithm.

Turning now to FIG. 6, a process for storing and querying conceptualindices in an inverted table is generally shown in accordance with anembodiment. At block 602, a query is received, and at block 604, data isaccessed that links text in a document to concepts in a concept graph.At block 606, a measure of closeness between the query and each of theconcepts is computed, and at block 608, a selected threshold number ofconcepts that are closest to the query are output.

In an embodiment, a method can include creating a conceptual invertedindex based on conceptual indices. The conceptual inverted indexincludes conceptual inverted index entries, each of which corresponds toa separate concept in a concept graph.

For each conceptual inverted index entry, the creating can include, withrespect to the concept corresponding to the conceptual inverted index,populating the conceptual inverted index entry with pointers todocuments selected from the conceptual index having likelihoods of beingrelated to the concept that are greater than a threshold value and thecorresponding likelihoods of the documents.

The method can also include receiving a query that includes one of theconcepts in the concept graph as a search term, searching the conceptualinverted index for the search term, and generating query results fromthe searching. In an embodiment, the query includes a plurality ofconcepts as the search term. The query results can include at least asubset of the pointers to documents. In an embodiment, the subsetincludes pointers to those documents having likelihoods greater than asecond threshold of being related to the concept included in the searchterm. In an embodiment, the query results can include a pointer to adocument that does not explicitly mention the search term.

Each of the conceptual indices can be associated with a correspondingone of the documents and includes a conceptual index entry for eachconcept in the concept graph, and each of the conceptual index entriesspecifies a value indicating a likelihood that the one of the documentsis related to the concept in the concept graph. In an embodiment, aconceptual index entry can indicate that a document is related to aconcept in the concept graph that is not mentioned in the document. Theconcept not mentioned in the document can be related to the conceptincluded in the query. In an embodiment, a document is related to aconcept in the concept graph if a concept extracted from the document isconnected to the concept in the concept graph via a path in the conceptgraph.

In an embodiment, each document is associated with a valid versionnumber and generating query results includes verifying that any pointersto documents correspond to documents that match the associated validversion numbers. Pointers to documents that do not match the associatedvalid version numbers can be removed from the conceptual inverted index.

The various embodiments described in here can be assembled to build afull end-to-end system that includes the ingestion of a collection ofdocuments into a system and the mechanisms for using this system througha user interface. It is important to note that even though fundamentallythe same system can be applied to different kinds of documents, thenature of the documents ingested into the system can dramatically changethe intended use for the system. For example, if the documents ingestedare news sources, then the system can be used for the exploration andrecommendation of news content. If the documents ingested are patents,then the resulting system can be used for prior art search. If thedocuments ingested are the descriptions of the expertise of a patentexaminer, then resulting system can be used for recommending whichpatent examiners to use for a given patent application. If the documentsingested are descriptions of the expertise of a body of researchers inan organization, the resulting system can be used to as an “expertiselocator” to help locate skills in an organization with the goal ofmaking its processes more efficient.

To describe the end-to-end system, it is assumed that there is acollection of documents for which conceptual exploration is desired, orfor which conceptual searches are desired, or from which a selection ismade in order to provide recommendations given some context or query.Throughout this description, numerous instances are described regardingthe capability of linking text to concepts in a concept graph. In onestep, the text of each document is passed through the module which linksthe document (and specific words within the document) to concepts in theconcept graph. Confidence scores about these linkages may be produced bythis step. The links and the possible confidence scores can constitutethe a priori information known about each document. Optionally, theconcept graph may be augmented prior to this step by analyzing thedocuments as a reference corpus. Extraction of candidate new conceptsfrom this reference corpus may happen by extracting noun phrases fromthis reference corpus. These noun phrases can then be regarded as astring of text for which a new concept definition may be considered. Theprocess of analyzing a string of text for this goal has been describedelsewhere herein.

In another step, the a priori information for each document can beanalyzed in conjunction with a concept graph in order to obtain arepresentation of the document in a concept space, obtaining aposteriori information for each document. This analysis is describedelsewhere herein.

After this step, the data produced so far is organized in a “conceptual”reverse index (or conceptual inverted index) which allows the fastretrieval of relevant documents given a conceptual query. In addition tothis reverse index, the system may produce an explanations index thatcan be used to help construct explanations to a user about the relevanceof a document to a conceptual query. The mechanisms for building andmaintaining these indices are described elsewhere in this embodiment. Asan alternate to producing an explanations index, the explanations can becomputed on the fly at the same time the query is made, as described inthis embodiment.

After the analysis is complete, the system exhibits a user interface inwhich an input can be entered. The input mechanism can allow the user tospecify one or more concepts from the concept graph, one or moredocuments (so that similar documents can be returned), or a combinationof both. Alternately, a plain string of text may be entered, which isthen passed to a mechanism that extracts concepts from it such as thetext to concept graph linker described in this embodiment.

After having input a query and a collection of concepts made availableto the system, which is referred to herein as a conceptual query, thesystem proceeds to lookup the concept based reverse index to finddocuments that are relevant to the one or more concepts in the query. Asthe reverse index is organized as a table that is looked up via singleconcept queries, multiple lookups are made and these lookups areaggregated to form a single presentation.

The documents returned are presented with an explanation of why they arerelevant to the query by emphasizing those concepts within the extractedconcepts in the document that are most relevant to the query and/orshowing related text containing those concepts. Additionally, thedocuments themselves maybe clustered in groups of documents that relateto the query in similar ways, as described earlier in this embodiment.The various concepts being emphasized may also be presented with ahyperlink; clicking on this hyperlink may have the effect of changingthe query to include the clicked concept.

The following text describes an end-to-end system for building anexpertise locator.

Described below are embodiments of system for creating and searching arepository of professional skills and interests that can be built usingembodiments of the semantic searching techniques described herein.Embodiments can provide a user interface for summarizing the relevanceof a document to a query. In addition, embodiments can provide a systemfor searching, recommending, and exploring documents through conceptualassociation.

In the example embodiment described below, the system is referred toherein as the “Researcher System” and it can serve as an external and/orinternal Web face for people and projects across a corporation's (orother group) research locations (or other types of locations).Embodiments can support a technique for automatically extracting andusing Wikipedia based concepts from the Researcher System's underlyingcontent store to support the finding of potential experts through thenovel forms of fuzzy matching described herein. In addition, theResearcher System can simplify content creation and linking in order tomake it easier for researchers (or other types of employees) to describetheir professional skills and interests and, in so doing, to make themeasier to find. Embodiments also include searches and user interfacesfor finding researchers that are conceptually related to a query despitethe absence of string matches between the query and the underlyingrepository.

In the example embodiment described herein, the Researcher Systemcontains descriptions of nearly 2,000 professionals, along with theirprojects, tens of thousands of their papers and patents, anddescriptions of their personal interests. Key aspects of the ResearcherSystem are described herein including how its design supports rapidauthoring, by individual researchers without central editorial control,of relatively coherent and richly linked content.

Also described herein is the use of the content in the Researcher Systemas a source of high value information about peoples' skills andinterests. An example shows how embodiments can be used to efficientlyindex this repository and map search queries to this index to findpeople with related skills and interests. Unlike systems relying onexplicit tag creation or fixed ontologies, embodiments include theability to automatically create something like tags using the conceptualspace provided by Wikipedia. Terms within search queries are mapped toconcepts within this space and the conceptual distance from the query toall the people in the repository is computed.

In embodiments of the Researcher System, individuals can create any HTMLcontent that they desire, from simple text to complex and interlinkedstructures, all of which is embedded at display time into a uniform setof templates. Dense networks of links between people and projects areencouraged both by example and semi-automatic and automatic linkcreation. Once authored and previewed, content is immediately publishedto the open Web without editorial review to support rapid iteration andprovide immediate reinforcement.

Embodiments of the Researcher System create characterizations of skillsand interests by automatic analysis of textual content. Thesecharacterizations make use of the vast collection of over five millionconcepts currently found within Wikipedia. Since each concept isextremely unlikely to be directly associated with any one person,embodiments make use of Wikipedia's link structure to find concepts nearthe concepts in a query. And, once a potentially relevant person isfound, the Researcher System can facilitate navigation to other relatedpeople, helping the searcher find people who might also be of interest.To accomplish this, the use of various components described in thisinvention may be employed. First, an automatic linking of the textdescribing the researchers to a concept graph derived from Wikipedia isperformed. The text of these researchers is processed to discover newconcepts that may be added to the concept graph. The information aboutthe automatic linking, which is regarded as as “a priori” informationabout the documents, in combination with the concept graph, is used toassociate each of the documents (representing a person) to each conceptin a concept graph, thus obtaining the representation of the document ina concept space.

In embodiments of the Researcher System, searches are based on conceptsexpressed using a search tool outlined below. Recall is substantiallyimproved by this technique since many concepts tend to lie near theconcept or concepts expressed in the query and many people tend to havemany associated concepts near these same concepts. This rich fuzzymatching almost always results in a set of relevant people beingreturned.

Embodiments of the Researcher System also provide, for ease of contentauthoring and immediate publishing of external content in order tofacilitate growth in the number of pages and increased page freshness.Embodiments also provide for links between people to enhance navigationin the Researcher System. Thus, once a searcher finds a person ofinterest, crosslinks to related people or projects are also found.Embodiments provide ease of crosslink creation (including automaticcrosslink creation where possible).

Turning now to FIGS. 7-10, primary views (user interfaces) of people andprojects in a deployment of the Researcher System are generally shown inaccordance with embodiments. Embodiments support both internal views(limited to those inside the corporation) and external views (visible toall).

FIG. 7 depicts an external view of a user interface 700 of a researcherprofile in accordance with an embodiment. As shown FIG. 7, photo, name,job role, location, and contact information are displayed at the topwith the email address and phone number being rendered in a way thatmakes scraping for spam purposes prohibitively expensive, if notimpossible. In FIG. 8, a user interface 800 illustrates a ProfileOverview, Publications Page, and Patents Page associated with theresearcher and are arrayed as a collection of tabbed pages. Other tabsmay be created to hold additional content, as shown in FIG. 10 (e.g.,Risk Perception”). The Profile can contain any HTML content, createdusing an editor shown in FIG. 11. In this fairly typical example, linksto related information, both information hosted in a repository of theResearcher System and that residing elsewhere on the open Web, areincluded in the Profile description.

Semi-automatically created links to related information are displayedalong the left hand column of the user interface 700 in FIG. 7. LinkedProject Pages are shown first, followed by Professional Interests (whichinclude both high-level disciplines such as Computer Science or Physics,and then topical interests such as Human Computer Interaction). Links toProfessional Associations (such as ACM) are shown last. This order canresult in putting links to conceptually nearby people in positions ofhigher prominence on the user interface.

Turning now to FIG. 8, the user interface 800 provides an external viewof a portion of this researcher's publication page in accordance with anembodiment. Publications can be added using a mechanism such as thatshown in FIG. 12 below and can be automatically sorted by year.Abstracts, when available, can also be automatically pulled fromexternal digital libraries. Links to these external documents, and linksto co-authors pages can also be automatically inserted. This supportquick navigation to both a paper and its collaborating authors.

Turning now to FIG. 9, a user interface 900 of an internal view of aportion of this researcher's Patents page is shown in accordance with anembodiment. Internal, in this case, means that the viewer is known (byvarious automatic means) to be an employee of the corporation. Note thatthe internal view of content shown in FIG. 9 differs from the externalview shown in FIG. 8 in two ways. First, content that is not yetexternalized is highlighted in red. This alerts the internal viewer thatthe content either needs additional work or simply needs to beexternalized using the editor. Second, an “Edit Patents Page” button isdisplayed if the internal viewer is also known to be the owner of thecontent. This makes access to the associated page editor only a singleclick away.

Project pages are rendered in a fairly similar fashion to people pages.FIG. 10 shows a user interface 1000 of a portion of one of thisresearcher's projects in accordance with an embodiment. The userinterface 1000 shown in FIG. 10 is an internal view of a project pageshowing the “join/edit group” button in accordance with an embodiment.The upper left of the page holds an optionally displayed photomontage ofall group members, allowing direct navigation to project collaborators.Tabs for the project Overview and an optional Publications page can alsobe provided by default. Other tabbed pages can also be added to provideadditional information (in this case, a page on Risk Perception).

FIG. 10 shows the internal view of this project, in particular, the viewprovided to an internal corporate employee who is not currently in theproject. As such, a “Join/Edit Group” button is displayed making ittrivially easy to join the project and have it automatically linked tothe viewer's personal page. This mechanism encourages the creation ofcrosslinks facilitating navigation and the understanding of explicitlyrelated people. A discussion of implicitly related people follows in alater section on concept-based people indexing and search.

FIGS. 11 and 12 show how content (beyond the links to related contentdiscussed above) can be authored in an embodiment of the ResearcherSystem. Turning now to FIG. 11, a user interface 1100 of an editor for aresearcher's profile page is generally shown in accordance with anembodiment. FIG. 11 shows the primary Profile editor that can berestricted to only being available to the particular researcher and/orto their designee. In the embodiment shown in FIG. 11, buttons topreview and externalize the page appear at the top of this page and canalso appear at the top of every content creation page. It should benoted that the editor shown in FIG. 11 has a close relationship to theexternal view. Content layout is essentially the same as in the viewwith only a few additional editor controls surrounding each majorsection. This helps ensure that the resulting view is easily visualizedduring editing. In addition, embodiments include the provision of both a“View Profile” button (for previewing) and an “Allow external viewing ofyour profile” checkbox that makes the content immediately visible on theopen Web. This can help motivate people to make quick updates ratherthan waiting for some major change or batching their smaller updatesbecause of a slow process of staging and vetting.

Still referring to the user interface 1100 of FIG. 11, edit controlsallow the researcher photo on the left, and the name, job role,location, and contact information on the right to be easily updated.When a page is first created, both the researcher's photo and thistop-level information can be automatically extracted from a corporatelightweight directory access protocol (LDAP) server to simplify pagecreation. Future editing sessions can update this information from theserver as needed unless the owner explicitly changes this content.Reasons for changing the job role from what is stored in the LDAP servercan include, for example, the addition of a university affiliation notcaptured in the LDAP record. Reasons for changing the location include,for example, specifying geographic information for a remote worker whois nominally associated with one of the global research laboratories butis actually physically located somewhere else.

Directly below the contact information in the user interface 1100 ofFIG. 11 is an edit control supporting the addition, deletion, renaming,and reordering of tabs (and their associated pages). In an embodiment,for cross-site consistency, Profile, Publications, and Patents pagescannot be moved or renamed. Publications and Patents pages can be madeoptional, however, but their use is encouraged both by the ease ofcreation afforded by the editor shown in FIG. 12 and by the provision ofindividual and manager views of citation counts that require a paper tobe indexed in the repository.

Below the tab control in the user interface 1100 of FIG. 11 is thewhat-you-see-is-what-you-get (WYSIWYG) editor for Profile page content.In an embodiment, the editor uses a tool such as TinyMCE with only a fewmodifications to limit the feature set and keep authored headingsappropriately situated within the final generated page structure. Tofurther ease content creation, the tool can also be used to convertpreviously authored content (say from a LinkedIn profile or a CV file)to HTML.

In the left column of the editor shown in the user interface 1100 ofFIG. 11 is the list of Projects, Professional Interests, andProfessional Associations, along with edit buttons for each. Projects,as noted above, are generally added by clicking on a “Join” button whenviewing the project itself. But it is also possible to see and selectfrom a list of all projects by clicking on the “Add/Change” button. Newprojects can also be created in this way. Similarly, the “Add/Change”button for Professional Associations allows the viewing and selectionfrom among more than 200 previously entered Professional Associations(along with the creation of new ones). Professional Interests differfrom these two categories in that it is only possible to select from afixed ontology of around 40 high-level disciplines and interest areas.Each discipline and interest area already has a page in the system thatis edited by the area owner to ensure it depicts the area (e.g., HumanComputer Interaction) appropriately.

Above these lists shown in the user interface 1100 of FIG. 11 is anoptional More Information section for any links that are thought by theauthor to be best displayed as a simple list. This section can alsoserve as an overflow area for any page tabs that cannot be renderedwithin a single row (a rare occurrence).

While there may be several other auxiliary editor components inembodiments of the Researcher System, only one more (the Publicationspage editor) will be discussed herein as it can be both a driver ofcontent for expertise location and a source of automatically generatedcrosslinks between people.

Referring now to FIG. 12, a user interface 1200 for a portion of aneditor for a researcher's publications page is generally shown inaccordance with an embodiment. Publications can be found in theResearcher System by the researcher clicking on the “Find Publicationsand Patents” button above the list of their publications (if any)selected during previous searches. A dialog allows the retrieval ofpublications from an external repository and the selection of those thatactually belong to them. Descriptions of papers can be changed if errorsare found or additional information needs to be added. Finally, papersnot yet indexed by the external repository can be added manually ifdesired.

An embodiment is not fully automated because it may require explicitselection from a list of possible papers, to account for the difficultyof unambiguously identifying authors in current external paperrepository implementations. However, this process takes only a fewminutes, even for researchers with a large number of publications andpatents. Once a paper or patent is identified, the automaticprovisioning of available abstracts, keywords, paper links, andcrosslinks between co-authors and co-inventors makes this modestinvestment worthwhile. Anther embodiment can be fully automated whenrepositories are used that have unique author identifiers and theprovision of suitable APIs by external paper repositories may eventuallymake this feasible.

Embodiments of the Researcher System utilize the semantic searchingtechnologies described herein. This can extend a rudimentary text searchcapability that allows finding people by portions of their name, theirnickname, and so on (along with a faceted search for finding people bydisciplines and laboratories), to allowing people to be found based onthe similarity between the concepts mentioned in their Researcher pagesand the conceptual content of a query.

An example of finding things conceptually related to a query usingembodiments of semantic searching techniques described herein follows.For example, a person may be trying to find someone in a researchpopulation who knows something about color blindness. FIG. 13A shows auser interface 1300 of the result of typing “color blindness” into asearch field. FIGS. 13B and 13C, as described previously depict howqueries that combine concepts and documents simultaneously are created.

Turning back to FIG. 13A, as characters are typed, the list of candidateconcepts shrinks until the concept of interest is found. Selecting thisstarts the search process.

Within a few seconds, a set of possible researchers is returned, rankedin decreasing order of conceptual closeness (described below). In orderto accomplish this retrieval task, reverse index technologies describedherein may be employed. A portion of this result set is shown in theuser interface 1400 of FIG. 14. There are things to note about thisresult set. First, examination of the Researcher System website showsthat none of these people explicitly mention “color blindness” in any oftheir content. Thus, a search based on string matching would not findthem. Second, the word cloud for each person, limited to their top sixmost highly related concepts in the embodiment shown in FIG. 14, allowsthe searcher to get a sense of both the kind of things in these Profilesthat are related to the query and the degree of relatedness of theseconcepts to the query. To generate this word cloud, techniques forsummarizing the relevance of a document to a query described herein maybe employed (e.g., in this expertise location use case embodiment,typically for every person approximately 50 concepts are extracted andonly the 6 most relevant to the query are displayed). In the case of thefirst two people this is shown, for example, by “web accessibility”being in a larger font than “input device” or “assistive technology”.Third, examining these particular word clouds makes it clear that twoquite different kinds of researchers have been returned as potentialexperts based on two quite different aspects of color blindness. Thefirst cluster includes two people with expertise in “web accessibility”and “low vision”, neither of which is exactly the same as colorblindness but is conceptually related. It is likely both people wouldeither know a lot about what can be done to accommodate color blindnessor would know an expert in this topic. The second cluster includes aresearcher with expertise in retinal cells. He too would either know afair bit about color blindness from the perspective of retinalphysiology or would likely know people who do. In an embodiment, theclusters described above are explicitly called out by the userinterface. To accomplish this task, the system takes the concept spacerepresentation of each document and then employs clustering algorithms(for example k-means) to create document clusters or groups ofdocuments. After obtaining these clusters, the associated documents aredisplayed in a user interface.

The task of summarizing the most relevant concepts in a document to aquery can be accomplished as follows. After a query is input, and a setof documents is returned through the use of the conceptual invertedindex, the concepts extracted for these documents are also retrieved.Next, a computation is made to estimate the degree to which the conceptor concepts in the query are relevant to each of the extracted conceptsfor all of the documents returned. This can be achieved by means of theMarkov chain techniques described earlier in this invention. Once thisestimate is obtained, the concepts within the documents can be ranked,and a desired number of them can be chosen for a word/concept clouddisplay. The displaying of these concepts can then utilize the relevancyinformation in order to affect their size on a display, or the color inwhich they are displayed. In addition, the extracted concepts not onlycould be displayed in a cloud, they could also be displayed in thecontext of the text that contains them, by rendering the text and theconcepts in differentiating ways.

It may be appreciated that the process described above can be performeddone at query time. The problem of summarizing the most relevantconcepts in a document to a query may also be done at document ingestiontime (essentially at the same time at which the conceptual invertedindex is being produced) by storing the contribution of each individualextracted concept from the document to a concept in the concept graph.

An embodiment of this procedure follows. Recall that a document isassociated with a conceptual index, which gives the likelihood that adocument is relevant for each concept in a concept graph. The way thisconceptual index is built is by computing the relevance of eachextracted concept in the document to each concept in the concept graph,and then aggregating these numbers into a single number (the aggregationmechanisms are described elsewhere herein). If in addition to theaggregated single number, the individual numbers are also stored, thenthere is an explanation of how individual extracted concepts in thedocument are relevant to each concept in the concept graph. Thisinformation, which can be regarded as an “explanations index” (e.g.,explanations index 114 shown in FIG. 1) can be used to produce thesummarized explanations in a user interface the same way the “querytime” algorithm described above does. However, note that the size ofthis possible explanations index is much larger than that of theconceptual index. A way to solve this problem is to store the summaryinformation for a document only for a subset of the concepts in thegraph that the document is most relevant to. One way to accomplish thisis to sort the likelihoods in a conceptual index for a document andselect the concepts that have the top W likelihood values, where W is aparameter that controls the size of this explanations index. Afterselecting those concepts, explanations can be stored for that documentonly for the selected concepts. These explanations are then retrieved atquery time for the documents for which we want to display a summary. Asbefore, excerpts of text from the document can also be highlighted withthe extracted concepts, adding differentiating visual elements todistinguish the ones that are the most relevant to the query.

In the descriptions above, both for the setting where summaries arecomputed at query time and when summaries are computed at indexing time,the embodiment described showed how to produce summaries for aconceptual query with a single concept. When there are two or moreconcepts in the query, additional steps described in here need to betaken. The first task is to either compute (when done at query time) orretrieve (when done at indexing time) the information of how each of theconcepts in the query relate to each of the extracted documents. Afterobtaining these independent pieces of information, the information mustbe merged. Let c1, c2, . . . c_Q be the concepts in the conceptualquery, let t1, t2, . . . t_E be the concepts extracted from thedocument, and suppose that the information for how relevant ti is for cjis given by some function f(ti, cj) (the function f(,) can be based onMarkov chain techniques as described elsewhere herein). At this point,the issue of how relevant an extracted concept ti (1<=i<=E) is to theconceptual query c1, c2, . . . c_Q is computed by aggregating the data[f(ti, c1), f(ti, c2), . . . , f(ti, c_Q)] into a single number. Manyaggregation techniques are possible. For example, the minimum of thesenumbers may be computed. Or the average of these numbers may becomputed. Or some transformation may be applied to the numbers beforethe aggregation; for example, since f(. , .) in this embodiment is alikelihood (a probability) then the log likelihood ratio of each numberin the list may be computed and then the numbers may be summed; the loglikelihood ratio for a likelihood p is given by log (p/(1-p)). At thispoint, a single number describing how relevant a concept in a documentis to a conceptual query is obtained and the rest of the steps follow asdescribed earlier.

Now, suppose that the topic the searcher was in interested in, but hadnot found a good way to yet express, involved the second, physiological,aspect of color blindness. The interface simplifies the expression ofthis refined query by allowing any of the terms in the word cloud to bedirectly clicked and submitted as the concept to search for next.

FIG. 15 shows a user interface 1500 with portion of the results returnedfrom the action of clicking “retinal ganglion cells” in the word cloudof Edward Daniels in FIG. 14 (Edward, of course, also being returned inthese new search results). It is clear that these are people working ondifferent, but potentially related, concepts including “neural encoding”and “neural networks”, either of which might be a bit closer to what asearcher is actually trying to find.

Embodiments of the Researcher System ingest and index the externalizedcontent within the Researcher System, including all profiles, secondarypages, and the titles of publications and patents. When content is addedor changed, it is automatically indexed within a few minutes of itsappearance.

An important aspect of this embodiment of the conceptual indexing systemis that the indexing of the corpus of data is assisted by an externaldata source (e.g., Wikipedia).

This happens in two stages. In the first stage, using the example ofWikipedia, the text generated by a researcher can be automaticallylinked to articles in Wikipedia in a process that has been previouslycalled “wikification”. The resulting links represent a form ofmachine-extracted knowledge that is the basis for the conceptualindexing technique. There are various published techniques for achievingthis stage. In embodiments of the Researcher System, a wikificationsystem is used that parses the content of a researcher's pages andphrases are matched against titles of Wikipedia articles. Terms near thematching phrase are checked against terms in the Wikipedia article andif they have good mutual information a match is registered. Thisapproach to disambiguation has the additional benefit of differentiatingbetween concepts that have the same descriptor (say “design” as shown inthe next section) but have quite different meanings.

In the second stage, an internal representation of related concepts canbe computed with which the researcher may be familiar. The informationretrieval literature provides multiple approaches for estimating this.An embodiment uses a variant of the class of techniques that computerelatedness between a concept and other concepts by computing the numberof short link paths between them. For each researcher the concepts foundin the first stage are augmented by using closely related concepts basedon this metric.

In addition to the conceptual relatedness metric described above,weights can be computed for the individual concepts associated with eachresearcher. The primary reason is that writing styles vary significantlyfrom researcher to researcher. A secondary reason is that the annotatorin the first stage sometimes makes mistakes and thus further de-noisingis needed. To illustrate the variability in writing styles, it can benoted that the length of the data for each researcher has a tremendousdynamic range, exhibiting a log-normal distribution. The log-normalbehavior implies that measuring expertise by simple linear accumulationof evidence is likely to disproportionally favor individuals with verylengthy descriptions. Embodiments of scoring mechanisms do give creditto researchers who have more content, however they also detect theverbosity in a researcher's description in order to adjust scoresappropriately.

A fuzzy matching approach as described herein using Wikipedia conceptscan be used to increase the likelihood of finding people related to aquery. In addition, embodiments can be used to increase the precision ofsearch results.

For example, consider a portion of the results returned from a searchfor people with expertise in “memory systems” as shown in the embodimentof a user interface 1600 shown in FIG. 16. Examining the publications,shown in the embodiment of a user interface 1700 shown in FIG. 17, ofthe first person in this set, it can be determined that he has publisheda paper on the “design” of memory systems and another on “designing” anaspect of a thin client.

Recalling that the system also ingests and indexes the titles ofpublications it might be assumed that a search for the term “design”would find this researcher. However, this might not be the case.

FIG. 18 shows user interface 1800 with a portion of the results that mayreturned from searching for the concept “design”. While different kindsof design-related researchers are returned (with people related tointeractive design and design methodologies being near the top), theaspect of design as applied to memory systems and thin clients does notrank as highly. As such, the precision of the search can be increased.

Also included in the results from the search for “color blindness” couldbe the researcher shown in the user interface 1900 of FIG. 19. He wassomewhat further down the ranked list of results, but not so far down asto be effectively invisible. At first glance it may be difficult tofigure out why this person was in included in the search results. But byturning on additional diagnostic information (which includes thecomputed distance metric at the top along with an icon at the bottomallowing evidence to be shown) it can be discovered that that thisperson was a co-inventor of the laser surgery technique that has come tobe known as LASIK. This is clearly something related to abnormal visionbut it is arguably too far away from color blindness to make this persona likely expert on that topic.

The system for finding people described in here can be augmented withsocial networking capabilities. In one embodiment, a mechanism can beadded for inputting into a system a “thought” which the system processesas a conceptual query. The documents in the system, as in the expertisefinding application, are documents that are representative of a person'sinterests and/or skills. Therefore, the system is capable of creating alist of people that may be interested in the thought input by assessingthe degree to which the thought is relevant to a person. The goal of thesystem is to enable an interaction between the initiator of the thoughtto the rest of the community in the system, for example in the form of achat (discussion) focused on the aforementioned thought. As such, thecloseness between the thought and the interests or skills of a person isonly one of multiple parameters that are considered in order to selectpeople that may be potentially interested in the discussion. Additionalparameters include whether a person is presently logged into the system,whether a person is presently or recently active in the system, whethera person has engaged with discussion requests or not, and the number ofdiscussions in which a person has engaged recently.

An application of this technology for resolving a different problem willnow be described. Suppose that it is desirable to build a system forrecommending news articles to a reader. Then in the documents that areinput to the system, news articles are included, which are thenautomatically linked to a concept graph by the system. In addition tothe article's text, the date in which the article was published isincluded as a piece of metadata. The end-to-end system as describedearlier is applied to this task. A search bar in a newspaper web page ore-reader device allows the user to input queries either by specifyingconcepts directly or by specifying a string of text. Recommendedarticles are then returned by the system by looking up the conceptualreverse index as described in this embodiment. In addition, recommendedarticles may also be given when a user is already reading an article, byanalyzing the article, extracting concepts in the article and thenfinding articles that are close to the article being read in aconceptual sense (as opposed to a text similarity sense, which is acommon way to do document similarity analysis). In addition, anavigation facility for news articles may be added by displayingconcepts related to articles with hyperlinks, clicking on the hyperlinksresults in redoing a conceptual search with the concept that wasclicked, and thus displaying new content. The sequence of articles andconcepts clicked can then be used to create a user profile to understandwhat interests the user in a conceptual sense. These concepts can thenbe used as contextual information to aid further article recommendationseven in the absence of any specific query from the user. The lists ofarticles returned by the system in the various cases described arefurther re-ranked to account for the date in which the article waswritten, so that more recent articles receive an increase in theirranking. This provides a hybrid ranking system considering conceptualrelevance and recency.

As described herein, a method for summarizing the relevance of adocument to a conceptual query can include receiving a conceptual querythat includes one or more concepts. Concepts extracted from the documentare accessed, and a degree to which the conceptual query is related toeach of the extracted concepts is computed (e.g. based on paths in aconcept graph connecting the concepts in the conceptual query to each ofthe extracted concepts). A summary can be created by selecting athreshold number of the concepts having a greatest degree of relation tothe conceptual query, and then output (e.g., to a user interface). Thesummary can include excerpts of text from documents highlightingextracted concepts that have a degree of relevance to the conceptualquery. Each of the concepts in the summary can be associated with ahyperlink and based on a user selecting the hyperlink, a new list ofsummaries related to the concept in the hyperlink can be output. Arelevance of an extracted concept to a conceptual query can besummarized by at least one of changing a font size of the extractedconcept and changing a color of the extracted concept.

In an embodiment, a least one concept in the summary is not a concept inthe conceptual query. The computing the degree of relation can includeiterating a Markov chain derived from the concept graph. The computingcan be performed at indexing time for conceptual queries comprised of asingle concept, and at least a subset of the results of the computingstored in an explanations index. Creating the summaries can includeretrieving the explanations index. The conceptual query can include twoor more concepts and creating a summary can be responsive to a degree ofrelation between each of the concepts in the conceptual query and eachof the extracted concepts.

As described herein, an embodiment for summarizing the relevance ofdocuments to a conceptual query can include: receiving the conceptualquery; accessing extracted concepts for each of the documents; computinga degree to which each of the documents are related to one another(e.g., as part of a clustering algorithm), the computing responsive topaths in the concept graph connecting the extracted concepts in onedocument to extracted concepts in another document; assigning thedocuments to one or more groups based on the computing, wherein a pairof documents having a first score that specifies a degree of relationare more likely to be in the same group than a pair of documents havinga second score specifying a degree of relation that is lower than thefirst score; and outputting results of the assigning.

Turning now to FIG. 20, a high-level block diagram of a question-answer(QA) framework 2000 where embodiments described herein can be utilizedis generally shown.

The QA framework 2000 can be implemented to generate an answer 2004 (anda confidence level associated with the answer) to a given question 2002.In an embodiment, general principles implemented by the framework 2000to generate answers 2004 to questions 2002 include massive parallelism,the use of many experts, pervasive confidence estimation, and theintegration of shallow and deep knowledge. In an embodiment, the QAframework 2000 shown in FIG. 20 is implemented by the Watson™ productfrom IBM.

The QA framework 2000 shown in FIG. 20 defines various stages ofanalysis in a processing pipeline. In an embodiment, each stage admitsmultiple implementations that can produce alternative results. At eachstage, alternatives can be independently pursued as part of a massivelyparallel computation. Embodiments of the framework 2000 don't assumethat any component perfectly understands the question 2002 and can justlook up the right answer 2004 in a database. Rather, many candidateanswers can be proposed by searching many different resources, on thebasis of different interpretations of the question (e.g., based on acategory of the question.) A commitment to any one answer is deferredwhile more and more evidence is gathered and analyzed for each answerand each alternative path through the system.

As shown in FIG. 20, the question and topic analysis 2010 is performedand used in question decomposition 2012. Hypotheses are generated by thehypothesis generation block 2014 which uses input from the questiondecomposition 2012, as well data obtained via a primary search 2016through the answer sources 2006 and candidate answer generation 2018 togenerate several hypotheses. Hypothesis and evidence scoring 2026 isthen performed for each hypothesis using evidence sources 2008 and caninclude answer scoring 2020, evidence retrieval 2022 and deep evidencescoring 2024.

A synthesis 2028 is performed of the results of the multiple hypothesisand evidence scorings 2026. Input to the synthesis 2028 can includeanswer scoring 2020, evidence retrieval 2022, and deep evidence scoring2024. Learned models 2030 can then be applied to the results of thesynthesis 2028 to generate a final confidence merging and ranking 2032.An answer 2004 (and a confidence level associated with the answer) isthen output.

Semantic analytics play a key role in information extraction by the QAframework 2000 shown in FIG. 20. Embodiments of the concept-drivenanalytics disclosed herein can be utilized by the QA framework 2000 toprovide relevant search and recommendation results, as well providingrich document exploration capabilities. A document (e.g., as question2002) can be processed for concept extraction (e.g., analysis 2010,decomposition 2012), and the extracted concepts may be used to determinerelationships against a concept space (e.g., answer sources 2006).Factors, such as the number of paths between concepts, as well as thelength of the paths thereof, can be used by the scoring mechanism 2020,to determine these relationships, and generate corresponding answers2004. In one embodiment, patterns of associations or relationships for agiven concept or group of concepts may be collected over time andstored, e.g., as models 2030, which are applied to subsequently derivedextracted concepts.

Referring now to FIG. 21, there is shown an embodiment of a processingsystem 2100 for implementing the teachings, including performing theprocesses (including the Researcher System), described herein. In thisembodiment, the processing system 2100 has one or more centralprocessing units (processors) 2101 a, 2101 b, 2101 c, etc. (collectivelyor generically referred to as processor(s) 2101). Processors 2101 arecoupled to system memory 2114 and various other components via a systembus 2113. Read only memory (ROM) 2102 is coupled to system bus 2113 andmay include a basic input/output system (BIOS), which controls certainbasic functions of the processing system 2100. The system memory 2114can include ROM 2102 and random access memory (RAM) 2110, which isread-write memory coupled to system bus 2113 for use by processors 2101.

FIG. 21 further depicts an input/output (I/O) adapter 2107 and a networkadapter 2106 coupled to the system bus 2113. I/O adapter 2107 may be asmall computer system interface (SCSI) adapter that communicates with ahard disk 2103 and/or tape storage drive 2105 or any other similarcomponent. I/O adapter 2107, hard disk 2103, and tape storage drive 2105are collectively referred to herein as mass storage 2104. Software 2120for execution on processing system 2100 may be stored in mass storageY2104. Network adapter 2106 interconnects system bus 2113 with anoutside network 2116 enabling processing system 2100 to communicate withother such systems. A screen (e.g., a display monitor) 2115 is connectedto system bus 2113 by display adapter 2112, which may include a graphicscontroller to improve the performance of graphics intensive applicationsand a video controller. In one embodiment, adapters 2107, 2106, and 2112may be connected to one or more I/O buses that are connected to systembus 2113 via an intermediate bus bridge (not shown). Suitable I/O busesfor connecting peripheral devices such as hard disk controllers, networkadapters, and graphics adapters typically include common protocols, suchas the Peripheral Component Interconnect (PCI). Additional input/outputdevices are shown as connected to system bus 2113 via user interfaceadapter 2108 and display adapter 2112. A keyboard 2109, mouse 2140, andspeaker 2111 can be interconnected to system bus 2113 via user interfaceadapter 2108, which may include, for example, a Super I/O chipintegrating multiple device adapters into a single integrated circuit.

Thus, as configured in FIG. 21, processing system 2100 includesprocessing capability in the form of processors 2101, and, storagecapability including system memory 2114 and mass storage 2104, inputmeans such as keyboard 2109 and mouse 2140, and output capabilityincluding speaker 2111 and display 2115. In one embodiment, a portion ofsystem memory 2114 and mass storage 2104 collectively store an operatingsystem such as the AIX® operating system from IBM Corporation tocoordinate the functions of the various components shown in FIG. 21.

Technical effects and benefits include the capability of analyzingdocuments conceptually instead of simply at the level of text matching,and using the result of this analysis to provide for more relevantsearch and recommendation results, as well providing for a rich documentexploration capability.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention. The computer readable storage medium can be atangible device that can retain and store instructions for use by aninstruction execution device.

The computer readable storage medium may be, for example, but is notlimited to, an electronic storage device, a magnetic storage device, anoptical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of onemore other features, integers, steps, operations, element components,and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A method for summarizing relevance of a documentto a conceptual query, the method comprising: receiving the conceptualquery, the conceptual query comprising one or more concepts in a conceptgraph; accessing concepts extracted from the document, the conceptsextracted from the document based on one or more differential testsperformed on portions of text in the document and at least a subset ofconcepts in the concept graph; computing a degree to which theconceptual query is related to each of the extracted concepts, thecomputing responsive to a metric that measures a relevance between theone or more concepts in the conceptual query and the each of theextracted concepts, the computing comprising using a transitionprobability matrix to perform a weighting of each link in the conceptgraph that connects the one or more concepts in the conceptual query tothe extracted concepts; creating a summary by selecting a thresholdnumber of the extracted concepts having a greatest degree of relation tothe conceptual query; and outputting the summary including the selectedthreshold number of extracted concepts.
 2. The method of claim 1,wherein at least one concept in the summary is not one of the one ormore concepts in the conceptual query.
 3. The method of claim 1, whereinthe computing further comprises iterating a Markov chain derived fromthe concept graph.
 4. The method of claim 1, wherein the metric utilizespaths in the concept graph connecting the concepts in the conceptualquery to each of the extracted concepts.
 5. The method of claim 1,wherein the summary is output to a user interface.
 6. The method ofclaim 5, wherein the each of the selected threshold number of conceptsin the summary is associated with a hyperlink via the user interface. 7.The method of claim 6, further comprising: receiving an indication thata user has selected the hyperlink from the user interface; and based onreceiving the indication, outputting a new list of summaries related tothe concepts in the hyperlink.
 8. The method of claim 5, where therelevance of an extracted concept to a conceptual query is summarized byat least one of changing a font size of the extracted concept andchanging a color of the extracted concept.
 9. The method of claim 5,wherein any concept in the selected number of concepts that also formspart of the conceptual query is not displayed by the user interface. 10.The method of claim 1, wherein the summary includes excerpts of textfrom documents highlighting extracted concepts that have a degree ofrelevance to the conceptual query.
 11. The method of claim 1, where thecomputing is performed at indexing time for conceptual queries comprisedof a single concept, and at least a subset of results of the computingare stored in an explanations index.
 12. The method of claim 11, wherethe creating a summary retrieves the explanations index.
 13. The methodof claim 1, wherein the conceptual query includes at least two concepts,and wherein the creating a summary is responsive to a degree of relationbetween each of the concepts in the conceptual query and each of theextracted concepts.
 14. A method for summarizing relevance of documentsto a conceptual query, the method comprising: receiving the conceptualquery, the conceptual query comprising a listing of one or more conceptsin a concept graph; accessing extracted concepts for each of thedocuments, the extracted concepts extracted from the each of thedocuments based on one or more differential tests performed on portionsof text in the each of the documents and at least a subset of conceptsin the concept graph; computing a degree to which each of the documentsare related to one another, the computing responsive to a metric thatmeasures a relevance between the extracted concepts in one document andextracted concepts in another document, the computing comprising using atransition probability matrix to perform a weighting of each link in theconcept graph that connects the extracted concepts in one document tothe extracted concepts in another document; assigning the documents toone or more groups based on the computing, wherein a pair of documentshaving a first score that specifies a degree of relation is more likelyto be in the same group than a pair of documents having a second scorespecifying a degree of relation that is lower than the first score; andoutputting results of the assigning including the one or more groups ofdocuments.
 15. The method of claim 14, where the computing the degree ofrelation among each of the documents is accomplished as part of aclustering algorithm.
 16. The method of claim 14, wherein the metricutilizes paths in the concept graph connecting the extracted concepts inone document to the extracted concepts in another document.